6 Replies Latest reply on May 6, 2016 7:57 AM by 2707430

    EBS PCP issues and Not working properly

    2707430

      we have enabled PCP on two MT node.

       

      below steps are done.

      APPLDCP = ON

      Autoconfig completed

      Primary and secondary nodes defined for each managers

      Internal managers and FNDFS  and service managers have been started on respective nodes.


      The problem - The fail-over and switch-over is not happening.


      1.. we shutdown Node1.. ICM failedover to Node 2 successfully. But ICM is not able to start the rest of concurrent managers on Node2 with below errors.

      version details : DB is 12.1.0.2 and EBS is 12.1.3



      Process monitor session started : 04-MAY-2016 22:31:42

      CONC-SM TNS FAIL

      Call to PingProcess failed for FNDOPP

      04-MAY-2016 22:31:43 - Node:(USABC2096), Service Manager:(FNDSM_USABC2096_ppzab) currently unreachable by TNS

      Starting FNDIM_USABC2096 Concurrent Manager        : 04-MAY-2016 22:31:43

      CONC-SM TNS FAIL

      Routine AFPEIM encountered an error while starting concurrent manager FNDIM_USABC2096 with library /u01/app/ebiz/ppzab/apps/apps_st/appl/fnd/12.0.0/bin/

      Check that your system has enough resources to start a concurrent manager process. Contact yo : 04-MAY-2016 22:31:43

      Starting PODAMGR Concurrent Manager                : 04-MAY-2016 22:31:43

      CONC-SM TNS FAIL

      Routine AFPEIM encountered an error while starting concurrent manager PODAMGR with library /u01/app/ebiz/ppzab/apps/apps_st/appl/po/12.0.0/bin/POXCON.

       

       

       

       



        • 1. Re: EBS PCP issues and Not working properly
          Bashar.

          Hi,

           

          Check the tnsnames.ora on both nodes.

          It should contain entries for both nodes in each file.

          How did you setup the primary and secondary nodes for the concurrent managers?

           

          Regards,

          Bashar

          • 2. Re: EBS PCP issues and Not working properly
            2707430

            The entries of both the hosts are fine.

            pinged self and other nodes. NO  issues on network and /etc/hosts.

             

            Primary and secondary hosts were thr front end (system administrator responsibility)

            Concurrent > Manager > Define screen, and set up the primary and secondary nodes)

            • 3. Re: EBS PCP issues and Not working properly
              Bashar.

              Please query fnd_concurrent_queues table and post the output.

               

              Regards,

              Bashar

              • 4. Re: EBS PCP issues and Not working properly
                2707430

                CONCURRENT_QUEUE_NAME          TARGET_NODE                    NODE_NAME                      NODE_NAME2                     P

                ------------------------------ ------------------------------ ------------------------------ ------------------------------ -

                FNDCRM                         USABC2096                      USABC2096                      USABC2097                      Y

                FNDICM                         USABC2096                      USABC2096                      USABC2097                      Y

                FNDSCH                         USABC2096                      USABC2096                      USABC2097                      Y

                FNDTMTST                                                      USABC2096                      USABC2097

                STANDARD                       USABC2096                      USABC2096                      USABC2097                      Y

                PODAMGR                        USABC2096                      USABC2096                      USABC2097                      Y

                RCVOLTM                        USABC2096                      USABC2096                      USABC2097                      Y

                PASMGR                         USABC2096                      USABC2096                      USABC2097                      Y

                WFMGSMS

                INVMGR                         USABC2096                      USABC2096                      USABC2097                      Y

                INVTMRPM                       USABC2096                      USABC2096                      USABC2097                      Y

                MRPMGR                         USABC2096                      USABC2096                      USABC2097                      N

                CRPINQMGR                      USABC2096                      USABC2096                      USABC2097                      N

                WFMGSMD

                WFALSNRSVC

                WFMLRSVC

                FTE_TXN_MANAGER

                WMSTAMGR

                XDP_MANAGER                    USABC2096                      USABC2096                      USABC2097

                Debug_Service

                C_AQCT_SVC

                IEU_SH_CS                      USABC2096                      USABC2096                      USABC2097                      Y

                IEU_WL_CS                      USABC2096                      USABC2096                      USABC2097                      Y

                XDP_CTRL_SVC

                XDP_Q_ORDER_SVC

                XDP_Q_WI_SVC

                XDP_Q_FA_SVC

                XDP_Q_FE_READY_SVC

                XDP_Q_EVENT_SVC

                XDP_Q_IN_MSG_SVC

                XDP_Q_TIMER_SVC

                XDP_APPL_SVC

                XDP_SMIT_SVC

                OKCCONCMGR                     USABC2096                      USABC2096                      USABC2097

                IEXCONMGR                      USABC2096                      USABC2096                      USABC2097

                OAMCOLMGR                      USABC2096                      USABC2096                      USABC2097                      Y

                FNDSM_USABC2097                USABC2097                      USABC2097

                FNDIM_USABC2097                USABC2097                      USABC2097                      USABC2096                      Y

                WFWSSVC

                AMSDMIN

                FNDCPOPP                       USABC2096                      USABC2096                      USABC2097

                OAMGCS_USABC2097                                              USABC2097

                FNDSM_USABC2096                USABC2096                      USABC2096

                FNDIM_USABC2096                USABC2096                      USABC2096                      USABC2097                      Y

                OAMGCS_USABC2096                                              USABC2096

                FFTM

                DownloadProcessorNormalMode

                DownloadProcessorMigrationMode

                • 5. Re: EBS PCP issues and Not working properly
                  Mohammed Ali A. Syed

                  Hi,

                  Please make sure you have the TNS entries of all the application nodes on each node.

                   

                  For FNDFS and FNDSM

                   

                  You should have 4 FNDFS entries for each application node.

                   

                  FNDFS_NODENAME

                  FNDFS_FQDNofNODENAME

                  FNDFS_<DB_NAME>_NODENAME

                  FNDFS_<DB_NAME>_FQDNofNODENAME

                   

                   

                  You should have 2 FNDSM entries for each application node.

                   

                  FNDSM_NODENAME_<DB_NAME>

                  FNDSM_FQDNofNODENAME_<DB_NAME>


                  And one entry for FNDFS_APPLTOP_localnode


                  and database node entries


                  Thanks

                  • 6. Re: EBS PCP issues and Not working properly
                    2707430

                    Applied the patches. 16735285 and 15981176

                    Here is the important observation on PCP.

                     

                    1. 1.. when u kill all process of Middle Tier  of node1 and node2 is able to ping Node1, switchover/failover happens within less than 10sec on node2 and vice versa. --- we are good on this

                     

                    1. 2.. If you shutdown the node1 for 10min..
                      • ICM gets started successfully on node2 within less than 2min.
                      • ICM of node2 tries reconnect to node1 and tries to start rest of the managers on node 2 but ICM is NOT able to start rest of the managers on node2.
                      • ICM of node2 is not able to release the locks of node1 since the node2 is not able to ping node1.
                      • The moment node1 comes online (Node 1 app listener still down), ICM immediately starts all managers on node 2 within 1-5sec.