3 Replies Latest reply on Nov 25, 2013 1:59 PM by Mike Kutz

    APEX 503 - Service Unavailable

    CraigG

      I had APEX 4 installed and working on Win 2008 R2 / Oracle 11.2 / Glassfish.

       

      About 4 days ago, I tried to login and instead of getting the login page, I got a 503 - Service Unavailable, as below:

       

      Request Path passes syntax validation

      Mapping request to database pool: PoolMap [_poolName=apex, _regex=null, _workspaceIdentifier=null, _failed=false, _lastUpdate=-1, _template=null, _type=REGEX]

      Applied database connection info

      Attempting to process with PL/SQL Gateway

      ==== Processing Request: ====

      GET /apex/f?p=4550:1:4548612085732

      ==== Headers in Request ====

      host: myhostname:8080

      connection: keep-alive

      accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8

      user-agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.57 Safari/537.36

      accept-encoding: gzip,deflate,sdch

      accept-language: en-US,en;q=0.8

      ==== Cookies in Request ====

      request parameter: p=4550:1:4548612085732

      isValidRequest(), procedure name:
      Validating: f
      ==== Request Processed ====
      PL/SQL Gateway is unavailable, will try to process as a RESTful Service
      Attempting to process as a RESTful Service
      Determining if request can be dispatched as a Tenanted RESTful Service
      Request path has one path segment, continuing processing
      No Tenant Principal established yet, continuing processing
      APEX_LISTENER pool exists, continuing processing
      Unable to process as a RESTful Service, reverting to Unavailable error reported by PL/SQL Gateway

       

      WebException [statusCode=503]
           at oracle.dbtools.rt.web.WebException.webException(WebException.java:326)
           at oracle.dbtools.rt.web.WebException.serviceUnavailable(WebException.java:306)
           at oracle.dbtools.rt.web.WebErrorResponse.serviceUnavailable(WebErrorResponse.java:167)
           at oracle.dbtools.rt.web.HttpEndpointBase.modApex(HttpEndpointBase.java:355)
           at oracle.dbtools.rt.web.HttpEndpointBase.service(HttpEndpointBase.java:130)
           at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
           at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1550)
           at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
           at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
           at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
           at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
           at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:161)
           at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:331)
           at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
           at com.sun.enterprise.v3.services.impl.ContainerMapper$AdapterCallable.call(ContainerMapper.java:317)
           at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:195)
           at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:860)
           at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:757)
           at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1056)
           at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:229)
           at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
           at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
           at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
           at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
           at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
           at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
           at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
           at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
           at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
           at java.lang.Thread.run(Thread.java:722)
      Caused by: oracle.dbtools.common.jdbc.ConnectionPoolException: The pool named: apex is not correctly configured, error: ORA-28000: the account is locked

           at oracle.dbtools.common.jdbc.ConnectionPoolException.badConfiguration(ConnectionPoolException.java:65)
           at oracle.dbtools.common.config.db.DatabaseConfig.badConfiguration(DatabaseConfig.java:166)
           at oracle.dbtools.common.config.db.DatabaseConfig.createPool(DatabaseConfig.java:187)
           at oracle.dbtools.common.config.db.DatabaseConfig.getConnection(DatabaseConfig.java:71)
           at oracle.dbtools.common.jdbc.ora.OraPrincipal.connection(OraPrincipal.java:69)
           at oracle.dbtools.apex.ModApexContext.getConnection(ModApexContext.java:372)
           at oracle.dbtools.apex.Procedure.getProcedure(Procedure.java:167)
           at oracle.dbtools.apex.OWA.validateProcedure(OWA.java:386)
           at oracle.dbtools.apex.security.Security.isValidRequest(Security.java:225)
           at oracle.dbtools.apex.ModApex.validateRequest(ModApex.java:266)
           at oracle.dbtools.apex.ModApex.doGet(ModApex.java:94)
           at oracle.dbtools.apex.ModApex.service(ModApex.java:302)
           at oracle.dbtools.rt.web.HttpEndpointBase.modApex(HttpEndpointBase.java:347)

       

      I checked the database and had found some passwords had expired, so I reset the passwords and unlocked the accounts:

      APEX_LISTENER

      SYSMAN

      DBSNMP

      APEX_PUBLIC_USER

      APEX_REST_PUBLIC_USER

       

      I rebuilt the apex.war file to set the password and deployed it through Glassfish.

       

      Retrying to login, I get the same error.

       

      I've noticed that the APEX_LISTENER user account keeps getting set back to LOCKED(TIMED) when viewed in OEM.

       

      I'm grasping at straws now and would appreciate any guidance to resolving this problem.

       

      I can provide more information if necessary (Glassfish log info).

       

      Thanks in advance.

       

      Craig

        • 1. Re: APEX 503 - Service Unavailable
          Mike Kutz

          LoL ... been there. done that. (but not with APEX)

           

          The Listener connects to the database with a configured name and password (I'm guessing yours is APEX_LISTENER)

          If connection fails, the Listener will try again... with the same password (... but you changed the password)

          Most DBs are configured so that:   after N failed attempts, the account will lock

          (It sounds like you reset the password prior to re-configuring the Listener)

          Fresh DB installs have the Default Profile set to expire after n days.  (You'll need to fix that or revisit this problem n days from now.)

           

          TO FIX

          1. stop the listener
          2. reset account
          3. configure listener
          4. retest the account using a tool (eg SQL*Plus or SQL Developer)
          5. start the listener
          6. wait, for 15min, and restest the acount

           

          LONG TERM FIX

          Database accounts have a "Default Profile" that defines such things as "Expire passwords after n days" and "Lock account after n Failures".

          Discuss with your DBA on how to handle this going forward as this is a DB thing, not an APEX thing.

          idea 1: (insecure) modify the Default Profile to set the 'expire password' to 'never'

          idea 2: create an "APEX Profile" with the 'expire password' set to 'never'

           

          other ideas

          You may need to look at the Parsing Schemas also.

           

          MK

          • 2. Re: APEX 503 - Service Unavailable
            CraigG

            Mike,

             

            Thank you for your reply.

             

            Using your steps as a guide, I was able to get APEX working again.  I do have a question - see item 3 below, please.

             

            I was close with what I had done the first time through this process.  For anyone else, here's what I did different based on your steps:

             

            1) This time, when I configured the apex.war, I did not specify a database. (On my previous attempt, I used my database name.  I guess I should have read the documentation more carefully.)  So the exact command was:

                 java -jar apex.war setup

            Just like the documentation.

             

            2) I deployed the new war file to Glassfish (no change there).

             

            3) I wasn't sure what you meant by stop and start the listener, so I took a stab at just stopping and starting the TNSListener service in Windows.  (Was that what you meant?)

             

            The end result, as I said, is that it works.

             

            To help reduce the likelihood of this occurring in the future, I create a new profile with unlimited password life and assigned the apex_listener and apex_public_user to that profile.

             

            Thank you again for the help.

             

            Best regards,

             

            Craig

            1 person found this helpful
            • 3. Re: APEX 503 - Service Unavailable
              Mike Kutz

              3) I wasn't sure what you meant by stop and start the listener, so I took a stab at just stopping and starting the TNSListener service in Windows.  (Was that what you meant?)

               

              The key thing is to disconnect the database connection between the APEX Listener and the database in order to prevent the APEX Listener from spamming the database with logon requests using a not-yet-up-to-date name&password.

               

              I normally stop the APEX Listener by shutdown the app server since APEX is the only thing we have running on that app server.

              Shutting down the database listener will work also.  But, for me, we have a few apps that directly connect to the database.  So, shutting down the database listener would disable their connections.  (ie a bad thing)

               

              MK