1 Reply Latest reply on Jan 25, 2017 11:39 AM by user5941356

    Failed / hanging validation of apex pool after several restarts of ords (3.0.9.348.07.16)

    user5941356

      Hi all,

       

      trying to figure out some strange behavior of our APEX applications,

      we saw that the validation of the APEX pool was hanging after restarting tomcat for severeal times.

       

      Apex 5.1 is installed on the Oracle DB (12.1.0.2 on Oracle Linux Server release 7.3).

      And yes, the APEX_PUBLIC_USER account is unlocked.

       

      To exclude tomcat issues, we reproduced the same behavior with ords standalone,

      i.e. we started ords (Java -jar ords.war standalone) and killed it (SIGINT, Ctrl+C) after a successful start.

       

      We were setting up new Linux (Debian 8.7, however we also tried Oracle Linux) machines on KVM and Xen.

      We utilized the OS included java (1.7.0_111, IcedTea 2.6.7) as well as the recommended one  included in the Oracle JDK (1.8.0_121).

       

      It seems the pool validation is hanging for 1/5 (Xen/KVM) minutes and running into a timeout.

      Here is the output (Xen in this case):

       

      INFO: Validating pool: |apex||

      Jan 20, 2017 8:22:37 AM

      SEVERE: The connection pool named: apex is not correctly configured, due to the following error(s): IO Error: Connection reset

      oracle.dbtools.common.jdbc.ConnectionPoolConfigurationException: The connection pool named: apex is not correctly configured, due to the following error(s): IO Error: Connection reset

              at oracle.dbtools.common.config.db.DatabasePools.validateConnection(DatabasePools.java:427)

              ...

      Caused by: java.net.SocketException: Connection reset

              at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115)

              ...

      Jan 20, 2017 8:22:37 AM

      SEVERE: *** Pool: |apex|| is not correctly configured and will be ignored ***

      Jan 20, 2017 8:22:37 AM

      INFO: Validating pool: |apex|al|

      Jan 20, 2017 8:22:37 AM

      INFO: Pool: |apex|al| is correctly configured

      Jan 20, 2017 8:22:37 AM

      INFO: Validating pool: |apex|pu|

      Jan 20, 2017 8:22:37 AM

      INFO: Pool: |apex|pu| is correctly configured

      Jan 20, 2017 8:22:37 AM

      INFO: Validating pool: |apex|rt|

      Jan 20, 2017 8:22:37 AM

      INFO: Pool: |apex|rt| is correctly configured

       

      We also tried on two hardware machines. On one of them it's always successful.

      On another machine, it shows the same behavior as for our Xen machine.

       

      As you can imagine, we don't really get what's going on.

      Anyone has an idea?

       

      Supplement:

      We used tcpdump to compare with successful pool validations.

      What we see in both cases:

      - TNS Request Connect

      - TNS Response, Resend

      - TNS Request Connect

      - TNS Response Accept

      - ... more identical TCP packets

      - TCP from host to DB with payload

        - APEX_PUBLIC_USER

        - AUTH_TERMINAL unknown

        - AUTH_PROGRAM_NM JDBC Thin Client

        - AUTH_MACHINE <host name>

        - AUTH_PID 1234

        - AUTH_SID root

      - TCP from DB to host with payload

        - AUTH_SESSKEY ...

        - AUTH_VFR_DATA ...

        - AUTH_GLOBALLY_UNIQUE_DBID ...

      If validation fails, this was the last packet.

      In the case of successful pool validation, we get the following packets:

      - TCP from host to DB, payload:

        - APEX_PUBLIC_USER_AUTH_PASSWORD ...

        - AUTH_TERMINAL unknown

        - AUTH_SESSKEY ...

        - ...

      - TCP from DB to host, payload:

        - AUTH_VERSION_STRING ...

        - AUTH_VERSION_SQL ...

        - ...

       

      We are grateful for every reference!  Thanks!