8 Replies Latest reply on Aug 3, 2016 7:28 AM by ora_tech

    CM turning into defunct state after startup

    2707430

      Hi,

       

      We have PCP enabled in our environment. Recently we just enabled TDE on our databases. Also there was a refresh of the instance as well. Oout of 20 managers, there are concurrent managers (transaction manager) are turning into defunct state immediately after CM startup.

       

      Any idea on this?

       

      abc_a  30434 21451  0 00:03 ?        00:00:00 [INCTM] <defunct>

      abc_a  30435 21451  0 00:03 ?        00:00:00 [RCVOLTM] <defunct>

      abc_a  30436 21451  0 00:03 ?        00:00:00 [POXCON] <defunct>

       

       

      Log:

       

      +---------------------------------------------------------------------------+

      Application Object Library: Concurrent Processing version 12.1

       

       

      Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.

       

       

      Server concurrent manager started (NODE=abcmv4XXX) : 31-JUL-2016 23:41:13

       

       

      +---------------------------------------------------------------------------+

      APPID 201  QID 222  CPID 8659

       

       

      31-JUL-2016 23:41:13 : Connecting account [APPS]

      31-JUL-2016 23:41:13

              Manager disconnecting, set PROCESS_STATUS_CODE to C

      31-JUL-2016 23:41:13

              Dropping current connection

      31-JUL-2016 23:41:13

              Manager reconnected

       

       

      Searching process (24255)

       

       

      Connected process 8659, session id 40657357, system pid 24255 after 0 tries

      31-JUL-2016 23:41:13

              Manager reconnected, updated fcp, status_code still A or C

      31-JUL-2016 23:41:13

              Requesting  mgr_lock

       

       

      AFPCSQ--  31-JUL-2016 23:41:13

              Manager process: cpid=(8659), manager=(201/222)

              Successfully received mgr_lock=(FNDCPLK_201_222_8659), lk_handle=(1073753912107375391246)

       

       

      31-JUL-2016 23:41:13 : Re-connected

       

       

      31-JUL-2016 23:41:13 : Initialized TM

       

       

      31-JUL-2016 23:41:13 : Read message

       

       

      31-JUL-2016 23:41:13 : Read message e_code [3]

       

       

      31-JUL-2016 23:41:13 : Unknown error

       

       

      31-JUL-2016 23:41:13 : Error occurred during shutdown of Transaction Manager.

        • 1. Re: CM turning into defunct state after startup
          Samrat Sinha

          Hi

           

          Can you upload the result of ps -ef | grep FND>>test.txt. And also see if this manager nodes are pointing to the correct nodes or not.

           

          Thanks

          • 2. Re: CM turning into defunct state after startup
            2707430

            Nodes are correct. I have checked thr back end and front end. Bounced the managers 2-3 times. Still the same issue.

            • 3. Re: CM turning into defunct state after startup
              2707430

              ps -ef | grep FND

              abc_a_a   6591     1  0 01:44 ?        00:00:00 /u01/app/ebiz/abc_a/apps/apps_st/appl/fnd/12.0.0/bin/FNDFSFNDFS (LOCAL=NO)

              abc_a_a   8864     1  0 01:50 ?        00:00:00 FNDIMON apps/ZGE2897975D667EFBAEF322EE157ADD405DED820C50CF8DC8AB604F27F7894BF881235FC14C6E723ED5C79B396B0BD0C74 FND FNDIM_USCMV4038 Y 60 i LOCK Y USCMV4038 8874

              abc_a_a   8869     1  0 01:50 ?        00:00:00 FNDCRM apps/ZG56C555939993BDF74B074706D6936D065200AEE736EDC59CE8E74BF989600837AF8BCDB3B40C75A915030CC35FCF1A5E FND FNDCRM Y 10 c LOCK Y USCMV4038 8882

              abc_a_a   8870     1  0 01:50 ?        00:00:00 FNDSCH                                                                                                         FND FNDSCH Y 10 s LOCK Y USCMV4038 8883

              abc_a_a   8871     1  0 01:50 ?        00:00:00 FNDLIBR                                                                                                                                                         

              abc_a_a   8872     1  0 01:50 ?        00:00:00 FNDLIBR                                                                                                                                                         

              abc_a_a   8873     1  0 01:50 ?        00:00:00 FNDLIBR                                                                                                                                                         

              abc_a_a   8875     1  0 01:50 ?        00:00:00 FNDLIBR                                                                                                                                                         

              abc_a_a   8877     1  0 01:50 ?        00:00:00 FNDLIBR                                                                                                                                                         

              abc_a_a   8878     1  0 01:50 ?        00:00:00 FNDLIBR                                                                                                                                                         

              abc_a_a   9565  9561  0 01:52 ?        00:00:00 FNDLIBR

              abc_a_a   9689     1  0 01:53 ?        00:00:00 FNDSM

              abc_a_a  19699  3890  0 02:09 pts/1    00:00:00 grep FND

               

               

               

              $ ps -ef |grep -i defunct

              abc_a_a  19757  9689  0 02:09 ?        00:00:00 [INCTM] <defunct>

              abc_a_a  19758  9689  0 02:09 ?        00:00:00 [RCVOLTM] <defunct>

              abc_a_a  19759  9689  0 02:09 ?        00:00:00 [POXCON] <defunct>

              abc_a_a  19798  3890  0 02:09 pts/1    00:00:00 grep -i defunct

              • 4. Re: CM turning into defunct state after startup
                Samrat Sinha

                Hi

                 

                What ICM log is saying? And what is the process_status_code for these managers under FND_CONCURRENT_PROCESSES table.

                 

                Did you also try running cmclean.sql? Any errors reported in alert log?

                 

                Thanks

                • 5. Re: CM turning into defunct state after startup
                  ;) ApPsMasTi ;)

                  Hi

                  please see below

                  How to Clean Nonexistent Nodes or IP Addresses From FND_NODES (Doc ID 260887.1)

                  Concurrent Processing - Command-Line Utility OR Recovery Wizard (Doc ID 134007.1)

                   

                   

                  AppsMasTi

                  sharing is Caring

                  • 6. Re: CM turning into defunct state after startup
                    2707430

                    Tried cleaning nodes and autoconfig.. No luck .

                     

                    Autconfig on second node is fine. But autconfig on second node throws below error.

                     

                    ERROR at line 1:

                    ORA-25207: enqueue failed, queue APPLSYS.WF_JAVA_DEFERRED is disabled from

                    enqueueing

                    ORA-06512: at "APPS.WF_EVENT", line 1181

                    ORA-06512: at "APPS.FND_WF_ENGINE", line 1025

                    ORA-06512: at "APPS.FND_PROFILE", line 3127

                    ORA-06512: at "APPS.FND_PROFILE", line 3577

                    ORA-06512: at "APPS.ADX_PRF_PKG", line 203

                    ORA-06512: at line 167

                    ORA-06512: at line 262

                    • 7. Re: CM turning into defunct state after startup
                      Samrat Sinha

                      Hi

                       

                      Please compile the invalids, run EXEC FND_CONC_CLONE.TARGET_CLEAN; and then run autoconfig in all the nodes starting from DB and post the results here.

                      • 8. Re: CM turning into defunct state after startup
                        ora_tech

                        Hi,

                         

                        As  suggested as a first step try to compile all invalid objects and ensure there are no invalid objects.

                         

                        thanks,

                        X A H E E R