2 Replies Latest reply on Dec 20, 2018 10:42 PM by user551320

    Ords standalone aborting after AIX server patches. uninstall not even working (FYI)


      Twas the week before Christmas and all thru the ORDS instance pair nothing was working not even a mouse.

      FYI in case anyone else hits a similar problem on AIX.


      I was in the process of populating a new Apex 18.2 / ORDS 18.3 instance on a new AIX server pairing. It was coming up fine on a Friday then by Wed of the next week it aborted with a hard java stack trace (of course I managed to lose those messages) and it refused to start up at all.  The stack trace msgs did not make sense since I had not touched any of the items it was referring to.


      Finally decided to uninstall it and start over from scratch. However, when the ords.war uninstall command asked me for the sys passwd it then kept telling me it could not even connect to the db. I did the normal quadruple verification of the usual suspects on connection failures and everything was fine. I finally had to rip everything out by hand and re-install from scratch. Still no luck.


      The only thing that had changed was that IT had applied the latest round of AIX patches during this timeframe. Somehow that was totally wiping out the normal functioning of the untouched ords.war file that had been working fine. Another aix server however was still functioning normally and it had recvd the same patch set.

      The only difference was that the still functioning aix server had everything (apex/ords/db) all on the same machine where as the failing server pairing had apex+db on 1 back-end server and ords on another front-end server.


      Presumably the new aix patch set somehow affected the 64bit version of Java pertaining to some part (JDBC related? TCP/IP domain sockets vs Unix domain sockets?) of the ords.war that let the front-end ORDS 18.3 on 1 server talk to the apex/db on the backend server.


      When I flipped the front-end ords.war back to 32bit Java 1.8 then everything started working just like it had been the previous week in 64bit mode.  But the other server (the one that had everything on 1 server) kept working fine in 64bit mode. I flipped back-n-forth between 32 and 64 bits and got re-creatable good/bad results.


      This was for Apex 18.2, ORDS 18.3, Oracle 12.2, Java 1.8 64bit vs 32bit on AIX

      java version "1.8.0_171"      (note this is after going back to 32bit hence ppc-32. the "good" server still says ppc-64)

      Java(TM) SE Runtime Environment (build - pap3280sr5fp15-20180502_01(SR5 FP15))

      IBM J9 VM (build 2.9, JRE 1.8.0 AIX ppc-32 20180425_385365 (JIT enabled, AOT enabled)

      OpenJ9   - a7ffbfe

      OMR      - a531219

      IBM      - 59ef3dc)

      JCL - 20180425_01 based on Oracle jdk8u171-b11