14 Replies Latest reply on Apr 10, 2017 8:12 PM by Dawaa Tamangg

    adoacorectl.sh: exiting with status 152 OPMN Managed Services

    Dawaa Tamangg

      Oacore Process managed by OPMN goes down at application tier with given below error at 9 AM twice or more a week and as check status of services oacore and http_server restarts daily at 9 AM and 1 PM .

       

      04/07/17-09:00:18 :: adoacorectl.sh: Starting OPMN managed OACORE OC4J instance

       

      opmnctl: starting opmn managed processes...

      ================================================================================

      opmn id=oser800001:6200

          0 of 1 processes started.

       

      ias-instance id=PROD_oser800001.oser800001.in.xyz.com

      ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      --------------------------------------------------------------------------------

      ias-component/process-type/process-set:

          default_group/oacore/default_group/

      Error

      --> Process (index=1,uid=824340971,pid=34266)

          time out while waiting for a managed process to start

          Log:

          /prodapp1/oracle/inst/apps/PROD_oser800001/logs/ora/10.1.3/opmn/default_group~oacore~default_group~1.log

       

      04/07/17-09:10:18 :: adoacorectl.sh: exiting with status 152

      04/07/17-09:10:34 :: adopmnctl.sh: Checking the status of Oracle Process Manager (OPMN)

       

      Processes in Instance: PROD_oser800001.oser800001.in.xyz.com

      ---------------------------------+--------------------+---------+----------+------------+----------+-----------+------

      ias-component                    | process-type       |     pid | status   |        uid |  memused |    uptime | ports

      ---------------------------------+--------------------+---------+----------+------------+----------+-----------+------

      OC4JGroup:default_group          | OC4J:oafm          |   29574 | Alive    |  824340965 |   118684 |  33:36:58 | rmi:25500,jms:24500,ajp:25000

      OC4JGroup:default_group          | OC4J:forms         |   29376 | Alive    |  824340964 |   129392 |  33:37:42 | rmi:20500,jms:23500,ajp:22000

      OC4JGroup:default_group          | OC4J:oacore        |     N/A | Down     |        N/A |      N/A |       N/A | N/A

      HTTP_Server                      | HTTP_Server        |   34214 | Alive    |  824340970 |    61888 |   0:10:03 | https1:4443,http1:8000

       

       

      As with the mechanism of OPMN ,opmn does pinging to component managed by it as follows.

       

       

      1. OS Process Check: Every 2 seconds OPMN queries the OS with the managed process id to see if it has terminated.

      2. Forward Ping: Periodically, 20 seconds by default, OPMN sends a ping message to the managed process and expects a result within 20 seconds.

      3. Reverse Ping: Every 20 seconds managed process sends OPMN a ping notification.

      If check #1 fails OPMN always attempts to restart the managed process.

      For check #2 and #3, if OPMN does not get a forward ping response, it flags the managed process as “DEAD” (i.e. unresponsive/unreachable).

        OPMN will continue to try and ping this "DEAD" process for max retry times. Max retry is the value of either of following two data elements:

      • reverseping-failed-ping-limit : when reverse pings are being received, OR

      • no-reverseping-failed-ping-limit : when reverse pings are also not being received (within the timeout period specified by reverseping-timeout data element).

      If any forward ping succeeds, the process state is set back to "ALIVE" (and the internal count towards max retry is set back to 0). If the forward ping fails consecutively for max retry times, then OPMN will attempt to stop and restart the process.

       

       

      This issue started to appear from 1 MARCH 2017 as follows in OPMN log as follows..

      17/03/01 09:10:17 [pm-requests] Request 258 Completed. Command: /start?process-type=oacore

      17/03/01 09:10:17 [pm-process] Stopping Process: default_group~oacore~default_group~1 (1243361365:12108)  <<===

      17/03/01 09:10:17 [libopmnoc4j] Process Stop Error: default_group~oacore~default_group~1 (1243361365:12108)  <<===

      17/03/01 09:10:17 [libopmnoc4j] Failed to construct stop command for proc: 1243361365   <<===

      17/03/01 09:10:17 [libopmnoc4j] Forcefully Terminating Process: default_group~oacore~default_group~1 (1243361365:12108) <<===

      17/03/01 09:10:19 [pm-process] Process Stopped: default_group~oacore~default_group~1 (1243361365:12108)

      17/03/01 09:53:57 [pm-requests] Request 267 Started.  Command: /start?process-type=oacore

      17/03/01 09:53:57 [pm-process] Starting Process: default_group~oacore~default_group~1 (1243361366:0)

      17/03/01 09:58:35 [pm-process] Process Alive: default_group~oacore~default_group~1 (1243361366:18795)  <<=== (Started mannually by me)

      17/03/01 09:58:35 [pm-requests] Request 267 Completed. Command: /start?process-type=oacore

       

       

      So far steps taken to resolve the issue:-

       

      1.. Middle tier application services has been bounced (stop and start).

      2.. Complete bounce of application tier has been done (stop and start)

      3.. Complete bounce of application tier has been done (stop and start) along with database bounce (stop and start) as given in oracle SR

      4.. Stoping middle tier application services ,removing contents of persistance folder and .opmndat file from states directory and persistance folder for OPMN managed component oafm,forms,oacore has been done. (After doing step 4 application worked for 1 day)

      5.. Oracle SR raised

       

      Points to be noted:-

       

      1.. Application tier (oser800001) node 1 was system boot  2.6.32-642.11.1 on Wed Feb 22 12:10 - 21:49 (22+09:38) without shutting down application tier gracefully  (not recommend or not done this way)

      2.. Application tier (oser800002) node 2 reboot   system boot  2.6.32-642.11.1. Thu Feb 23 07:33 - 21:40 (21+14:07)  without shutting down application tier gracefully  (not recommend or not done this way)

      3.. Database tier  (oser60001a) system boot  2.6.32-642.11.1. Mon Feb 27 05:34 - 21:32 (17+15:58) (Application tier both node 1 and node 2 running but database server was reboot without shutting down Database server gracefully and not recommend or not done this way

      4.. Reason to reboot is said to be patching of linux server for eg glibc-2.12.1.149.el6.x86_64

       

      JVM size defined 1GB

       

      Please suggest.

      Regards

      Dawa Tamang

        • 1. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
          mdtaylor

          Hi Dawa,

           

          First thing I would try is shutdown the applications, edit /etc/services on the application tier and comment out a line with 6200 and reboot the server.

           

          If that does not resolve it, See if there are any errors in the following files:

           

          /prodapp1/oracle/inst/apps/PROD_oser800001/logs/ora/10.1.3/opmn/default_group~oacore~default_group~1.log

          /prodapp1/oracle/inst/apps/PROD_oser800001/logs/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log

          /prodapp1/oracle/inst/apps/PROD_oser800001/logs/ora/10.1.3/j2ee/oacore/oacore_default_group_1/server.log

           

          Regards,

          Michael

          • 2. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
            mdtaylor

            I would also increase the Xmx and Xms to 2048M and possibly increase the oa_core nprocs to more than 1 if you have more than 100 users logging in.

             

            Change the following line in your $CONTEXT_FILE and run autoconfig:

             

            <oacore_nprocs oa_var="s_oacore_nprocs">1</oacore_nprocs>

            • 3. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
              Dawaa Tamangg

              HI mdtaylor,

               

              lm-x            6200/tcp                # LM-X License Manager by X-Formation

              lm-x            6200/udp                # LM-X License Manager by X-Formation

               

              I do not understand what is point of commenting line no 6200. As i found this at /etc/services. Under oacore log this is written when oacore stops

               

              --------

              17/04/07 09:00:08 Stop process

              --------

              69891.794: [GC 279568K->144589K(506880K), 0.0254610 secs]

               

               

              --------

              17/04/07 09:00:18 Start process

              --------

              17/04/07 09:00:18 WARNING: ApplicationLogManager is not installed, may result in loader leaks. Set -Djava.util.logging.manager=oracle.classloader.util.ApplicationLogManager

               

               

              --------

              17/04/07 09:11:19 Start process

              --------

              17/04/07 09:11:19 WARNING: ApplicationLogManager is not installed, may result in loader leaks. Set -Djava.util.logging.manager=oracle.classloader.util.ApplicationLogManager

              • 4. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
                Vishweshwara

                Hi Tamangg,

                 

                Clear the TLD cache:

                1)stop all middle tier services

                2)Delete/backup all the files under directory:

                $COMMON_TOP/_TldCache
                $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence

                cd $COMMON_TOP
                mv _TldCache _TldCache_bkp

                cd $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/
                mv persistence persistence_bkp

                3)start all middle tier services

                 

                Cheers,

                Vishweshwara Adiga

                • 5. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
                  Dawaa Tamangg

                  Hi Vishweshwara,

                   

                  I have done this i cleared the cache through front end. and have done removing of persistence folder and .opmndat file .

                   

                  I have problem of oacore process going down at 9 AM exactly couple of time in week and both oacore and http services managed by OPMN is restarting daily at 9 AM am 1 PM.

                   

                  Regards

                  Dawa Tamang

                  • 6. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
                    Vishweshwara

                    Hi Tamangg,

                     

                    Is it going down same day and time on every week ? If yes .. Please for crontab entries , anything is scheduled ?

                     

                     

                    Thanks

                    • 7. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
                      Dawaa Tamangg

                      H Vishwera,

                       

                      It oacore goes couple of time in week and some time daily .  I had to start manually.  What do you mean by  anything is scheduled ?

                       

                      Points to be noted:-

                       

                      1.. Application tier (oser800001) node 1 was system boot  2.6.32-642.11.1 on Wed Feb 22 12:10 - 21:49 (22+09:38) without shutting down application tier gracefully  (not recommend or not done this way)

                      2.. Application tier (oser800002) node 2 reboot   system boot  2.6.32-642.11.1. Thu Feb 23 07:33 - 21:40 (21+14:07)  without shutting down application tier gracefully  (not recommend or not done this way)

                      3.. Database tier  (oser60001a) system boot  2.6.32-642.11.1. Mon Feb 27 05:34 - 21:32 (17+15:58) (Application tier both node 1 and node 2 running but database server was reboot without shutting down Database server gracefully and not recommend or not done this way

                      4.. Reason to reboot is said to be patching of linux server for eg glibc-2.12.1.149.el6.x86_64

                       

                      Do you think patch update will cause oacore going down ..

                       

                      Regards

                      Dawaa Tamangg

                      • 8. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
                        mdtaylor

                        I asked you to comment out the 6200 lines from /etc/services due to the following lines from your log:

                         

                        opmn id=oser800001:6200

                            0 of 1 processes started.

                         

                        You will likely need to reboot the application server to make the change take effect.

                         

                        Regards,

                        Michael

                        • 9. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
                          Dawaa Tamangg

                          HI Michael,

                           

                          I believe  i am close enough to resolve the issue. I checked in both application tier node 1 and node 2 but port number allocated is 6500. Whereas default port range is 6200 where OPMN is trying to ping or restart oacore process. Why OPMN is pinging 6200 why not 6500 if it assigned.

                           

                          Thanks for taking close to solution i am one step behind it.

                           

                          Regards

                          Dawaa Tamangg

                          • 10. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
                            mdtaylor

                            Hi Dawaa,

                             

                            In your $CONTEXT_FILE, what is the value of s_ons_reportport?  It should be between 6200 and 6300.

                             

                            Regards,

                            Michael

                            • 11. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
                              Dawaa Tamangg

                              Hi Michael,

                               

                              Here it is :-

                               

                              <oa_ports>

                                              <PORT_POOL oa_var="s_port_pool">-1</PORT_POOL>

                                              <dbport oa_var="s_dbport" oa_type="EXT_PORT" base="1521" step="1" range="-1" label="Database Port">1521</dbport>

                                              <rpc_port oa_var="s_rpcport" oa_type="PORT" base="1626" step="1" range="-1" label="RPC Port">1626</rpc_port>

                                              <web_ssl_port oa_var="s_webssl_port" oa_type="PORT" base="4443" step="1" range="-1" label="Web SSL Port">4443</web_ssl_port>

                                              <ons_localport oa_var="s_ons_localport" oa_type="PORT" base="6100" step="1" range="-1" label="ONS Local Port">6100</ons_localport>

                                              <ons_remoteport oa_var="s_ons_remoteport" oa_type="PORT" base="6200" step="1" range="-1" label="ONS Remote Port">6200</ons_remoteport>

                                              <ons_requestport oa_var="s_ons_requestport" oa_type="PORT" base="6500" step="1" range="-1" label="ONS Request Port">6500</ons_requestport>

                                              <web_port oa_var="s_webport" oa_type="PORT" base="8000" step="1" range="-1" label="Web Listener Port">8000</web_port>

                                              <activewebport oa_var="s_active_webport" oa_type="DUP_PORT" base="8000" step="1" range="-1" label="Active Web Port">8000</activewebport>

                                              <forms_port oa_var="s_formsport" oa_type="PORT" base="9000" step="1" range="-1" label="Forms Port">9000</forms_port>

                                              <met_data_port oa_var="s_metdataport" oa_type="PORT" base="9100" step="1" range="-1" label="Metrics Server Data Port">9100</met_data_port>

                                              <met_req_port oa_var="s_metreqport" oa_type="PORT" base="9200" step="1" range="-1" label="Metrics Server Request Port">9200</met_req_port>

                                              <jtf_fulfillment_port oa_var="s_jtfuf_port" oa_type="PORT" base="9300" step="1" range="-1" label="JTF Fulfillment Server Port">9300</jtf_fulfillment_por

                               

                              Regards

                              Dawaa Tamangg

                              • 12. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
                                Dawaa Tamangg

                                Hi Michale,

                                 

                                Correct me if i am wrong . By OPMN definition opmn is running at port number s_ons_requestport 6500 given in "CONTEXT FILE" and OPMN keeps on checking its component

                                through port number s_ons_remoteport 6200 for following reasons:--

                                 

                                a) restart-on-death

                                b) ping timeout

                                c) ping interval

                                d) reverse-ping timeout

                                 

                                In my case port number 6200 is occupied by lm-x 6200/tcp  # LM-X License Manager by X-Formation or this process occupies this port number and when OPMN tried to ping and check port numbe rthrough this port was not free so it killed and tried to restart its managed process "oacore" leading OPMN to timeout "opmn id=oser800001:6200" leading oacore to DOWN as given in oacore log file.

                                 

                                Commenting out port number 6200 in /etc/services will disable this process lm-x 6200/tcp  # LM-X License Manager by X-Formation and available for OPMN to ping its component to check.

                                 

                                Thank you very much for guidance.

                                Regards

                                Dawaa Tamangg

                                • 13. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
                                  mdtaylor

                                  Did you run rapid install or clone to create the application tiers?  If something was listening on 6200 and you used port pool zero, ie 8000, then rapidwiz should have prevented you from installing until all of the ports were free.

                                   

                                  You should ensure all of the ONS ports are free (6100, 6200, 6500).  Check

                                   

                                  netstat -anp | grep 6100

                                  netstat -anp | grep 6200

                                  netstat -anp | grep 6500

                                   

                                  If any ports are occupied comment out of /etc/services.  If that doesn't resolve it, you will have to figure out what service is starting on those ports with chkconfig --list and disable them.

                                   

                                  I am pretty sure you are not using LM-X License Manager on an Oracle EBS server.  If you are, maybe you can reconfigure it to use a different port.

                                   

                                  In general, I prefer to avoid installing EBS on port pool 0 due to tendencies to experience port conflicts.

                                   

                                  Regards,

                                  Michael

                                  • 14. Re: adoacorectl.sh: exiting with status 152 OPMN Managed Services
                                    Dawaa Tamangg

                                    HI Michael ,

                                     

                                    I do not know how it was done whether rapid wizard or cloning.  All application tier and database tier were reboot by server team without shutting down gracefully. later on the day  i found application was down  . I had to start one by one even database. On request i came to know all application and database server were patch like glibc and others. .

                                     

                                    Issue with oacore started to happen after this .event.

                                     

                                    Regards

                                    Dawaa Tamangg