1 2 Previous Next 15 Replies Latest reply on Dec 9, 2018 1:07 PM by Beauty_and_dBest

    Concurrent Managers reset to zero?

    Beauty_and_dBest

      EBS 12.2.6

      OL6

       

       

      Hi ALL,

       

      After applying some fixes and then run autoconfig, we started our apps-tier with status 0.

      But to my surprise, all concurrent managers are reset to zero? (actual=0, target=0) 

       

      What causes the values to reset to zero?

       

       

      Please help how to set the correct values back.

       

       

      Kind regards,

      jc

        • 1. Re: Concurrent Managers reset to zero?
          mdtaylor

          Hi jc,

           

          I am not sure how this happened.  Please turn off the concurrent managers again with $ADMIN_SCRIPTS_HOME/adcmctl.sh stop apps/apps_pw

           

          Then ps -ef | grep FND on the concurrent manager server and make sure no FNDLIBR, or FNDSM processes are running.

           

          Here are what my 12.2 concurrent managers look like.

           

          Screen Shot 2018-12-02 at 10.14.06 AM.png

          You can reset with the following SQL:

           

          update fnd_concurrent_queues set min_processes=1, target_processes=1

          where concurrent_queue_name in ('CRPINQMGR','FNDCPOPP','FNDCRM','INVMGR','INVTMRPM','RCVOLTM','WFALSNRSVC','WFMLRSVC','WFWSSVC');

           

          update fnd_concurrent_queues set min_processes=10, target_processes=10

          where concurrent_queue_name ='STANDARD';

           

          Then once you restart the concurrent managers, you should be able to activate any other managers and set them the way you have them in another dev/test environment.

           

          If that doesn't work, use $FND_TOP/cpadmin.sh then

           

          1.    Administer Concurrent Managers

           

             2.    Clean concurrent processing tables

                    - Ensure concurrent processing tables are cleaned and reset for ICM startup

           

          Regards,

          Michael

          • 2. Re: Concurrent Managers reset to zero?
            Beauty_and_dBest

            Thanks Michael,

             

            Is cmclean.sql applicable here? I used to run it in our 12.1.3 EBS instance.

             

            Right now, there is no FNDLIBR, or FNDSM, or FND*   running, when I do > ps -ef|grep FND

             

            But I will try the action plan above, ...brb.

            • 3. Re: Concurrent Managers reset to zero?
              Beauty_and_dBest

              Hi Michael,

               

              I tried to update it but still the issue persist

               

              Concurrent Manager's Name                 TARGET Processes ACTUAL Processes

              ---------------------------------------------- ---------------- ----------------

              C AQCART Service                                         0           0
              Output Post Processor                                    0           4
              Conflict Resolution Manager                              0           1
              Internal Manager                                         0           1
              Scheduler/Prereleaser Manager                            0           1
              OAM Metrics Collection Manager                           0           0
              Standard Manager                                         0           0
              Workflow Agent Listener Service                          0           0
              Workflow Mailer Service                                  0           0
              Workflow Document Web Services Service                   0           0
              PO Document Approval Manager                             0           0
              Receiving Transaction Manager                            0           0
              PA Streamline Manager                                    0           0
              WMS Task Archiving Manager                               0           0
              Inventory Manager                                        0           0
              INV Remote Procedure Manager                             0           0
              Marketing Data Mining Manager                            0           0
              SFM Application Monitoring Service                       0           0
              SFM Controller Service                                   0           0
              SFM Event Manager Queue Service                          0           0
              SFM Fulfillment Actions Queue Service                    0           0
              SFM Fulfillment Element Ready Queue Service              0           0
              SFM Inbound Messages Queue Service                       0           0
              SFM Order Queue Service                                  0           0
              SFM Timer Queue Service                                  0           0
              SFM Work Item Queue Service                              0           0
              SFM SM Interface Test Service                            0           0
              Session History Cleanup                                  0           0
              UWQ Worklist Items Release for Crashed session           0           0
              Transportation Manager                                   0           0
              FastFormula Transaction Manager                          0           0

               

              31 rows selected.

               

              The original source PROD has below list:

               

              Concurrent Manager's Name                  TARGET Processes ACTUAL Processes

              ---------------------------------------------- ---------------- ----------------

              Conflict Resolution Manager                               1            1
              Internal Manager                                          1            1
              Scheduler/Prereleaser Manager                             1            1
              Standard Manager                                        100          100
              PO Document Approval Manager                              1            1
              Receiving Transaction Manager                             1            1
              PA Streamline Manager                                     1            1
              Inventory Manager                                         1            1
              INV Remote Procedure Manager                              1            1
              Workflow Agent Listener Service                           1            1
              Workflow Mailer Service                                   1            1
              Transportation Manager                                    1            1
              WMS Task Archiving Manager                                0            0
              C AQCART Service                                          0            0
              Session History Cleanup                                   1            1
              UWQ Worklist Items Release for Crashed session            1            1
              SFM Controller Service                                    0            0
              SFM Order Queue Service                                   0            0
              SFM Work Item Queue Service                               0            0
              SFM Fulfillment Actions Queue Service                     0            0
              SFM Fulfillment Element Ready Queue Service               0            0
              SFM Event Manager Queue Service                           0            0
              SFM Inbound Messages Queue Service                        0            0
              SFM Timer Queue Service                                   0            0
              SFM Application Monitoring Service                        1            1
              SFM SM Interface Test Service                             0            0
              OAM Metrics Collection Manager                            1            1
              Workflow Document Web Services Service                    1            1
              Marketing Data Mining Manager                             0            0
              Output Post Processor                                     4            4
              FastFormula Transaction Manager                           0            0

               

              31 rows selected.

               

               

              Kind regards,

              • 4. Re: Concurrent Managers reset to zero?
                Kanda-Oracle

                Hi JC

                 

                1. Check whether the Apps Listener is up and running.

                 

                  $ adalnctl.sh status

                 

                2.  Please restart concurrent manager with diag=y enabled.

                 

                $ adcmctl.sh start apps/<passwd> diag=y

                 

                3. Review .mgr .txt file under $APPLCSF/$APPLLOG

                 

                4. We need act as per the errors in the .mgr file

                 

                Thanks!

                 

                N Kandasamy

                • 5. Re: Concurrent Managers reset to zero?
                  Beauty_and_dBest

                  Hi Kanda,

                   

                  [applmgr@oraapps-2 ~]$ adalnctl.sh status

                   

                  adalnctl.sh version 120.3.12020000.4

                   

                  Checking status for listener process APPS_PPAPROD.

                   

                  LSNRCTL for Linux: Version 10.1.0.5.0 - Production on 03-DEC-2018 11:32:24

                   

                  Copyright (c) 1991, 2004, Oracle.  All rights reserved.

                   

                  Connecting to (ADDRESS=(PROTOCOL=TCP)(Host=oraapps-2)(Port=1626))

                  STATUS of the LISTENER

                  ------------------------

                  Alias                     APPS_PPAPROD

                  Version                   TNSLSNR for Linux: Version 10.1.0.5.0 - Production

                  Start Date                03-DEC-2018 10:26:45

                  Uptime                    0 days 1 hr. 5 min. 39 sec

                  Trace Level               off

                  Security                  ON: Local OS Authentication

                  SNMP                      OFF

                  Listener Parameter File   /u01/oracle/PPAPROD/fs1/inst/apps/PPAPROD_oraapps-2/ora/10.1.2/network/admin/listener.ora

                  Listener Log File         /u01/oracle/PPAPROD/fs1/inst/apps/PPAPROD_oraapps-2/logs/ora/10.1.2/network/apps_ppaprod.            log

                  Listening Endpoints Summary...

                    (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=oraapps-2.prompt.ppa)(PORT=1626)))

                  Services Summary...

                  Service "FNDFS" has 1 instance(s).

                    Instance "FNDFS", status UNKNOWN, has 1 handler(s) for this service...

                  Service "FNDSM" has 1 instance(s).

                    Instance "FNDSM", status UNKNOWN, has 1 handler(s) for this service...

                  The command completed successfully

                   

                  adalnctl.sh: exiting with status 0

                   

                   

                  adalnctl.sh: check the logfile /u01/oracle/PPAPROD/fs1/inst/apps/PPAPROD_oraapps-2/logs/appl/admin/log/adalnctl.txt             for more information ...

                   

                  [applmgr@oraapps-2 ~]$

                   

                   

                  [applmgr@oraapps-2 ~]$ adcmctl.sh start apps/apps diag=y

                   

                  You are running adcmctl.sh version 120.19.12020000.7

                   

                  Starting concurrent manager for PPAPROD ...

                  Starting PPAPROD_1203@PPAPROD Internal Concurrent Manager

                  Default printer is noprint

                   

                  adcmctl.sh: exiting with status 0

                   

                   

                  adcmctl.sh: check the logfile /u01/oracle/PPAPROD/fs1/inst/apps/PPAPROD_oraapps-2/logs/appl/admin/log/adcmctl.txt for more information ...

                   

                  [applmgr@oraapps-2 ~]$

                  • 6. Re: Concurrent Managers reset to zero?
                    mdtaylor

                    Hi jc,

                     

                    You can shutdown the concurrent managers again and run your old cmclean.sql.  It is not recommended anymore for 12.2.5 and above.

                     

                    Regards,

                    Michael

                    • 7. Re: Concurrent Managers reset to zero?
                      Beauty_and_dBest

                      Hi ALL,

                       

                      Below is the $APPLCSF/$APPLLOG

                       

                       

                      ========================================================================

                      Starting PPAPROD_1203@PPAPROD Internal Concurrent Manager -- shell process ID 28199

                       

                                logfile=/u01/oracle/PPAPROD/fs_ne/inst/PPAPROD_oraapps-2/logs/appl/conc/log/PPAPROD_1203.mgr

                                PRINTER=noprint

                                 mailto=applmgr

                                restart=N

                                   diag=N

                                  sleep=30

                                   pmon=4

                                 quesiz=1

                                Reviver is ENABLED

                       

                      sh: line 68: 28204 Segmentation fault      FNDLIBR FND CPMGR "FNDCPMBR sysmgr=\"\" sleep=30 pmon=4 quesiz=1 diag=N logfile=/u01/oracle/PPAPROD/fs_ne/inst/PPAPROD_oraapps-2/logs/appl/conc/log/PPAPROD_1203.mgr "  <<STOP

                      $unpw

                      STOP

                       

                      The PPAPROD_1203@PPAPROD internal concurrent manager has terminated with status 139 - giving up.

                       

                      ========================================================================

                      Starting PPAPROD_1203@PPAPROD Internal Concurrent Manager -- shell process ID 30572

                       

                                logfile=/u01/oracle/PPAPROD/fs_ne/inst/PPAPROD_oraapps-2/logs/appl/conc/log/PPAPROD_1203.mgr

                                PRINTER=noprint

                                 mailto=applmgr

                                restart=N

                                   diag=y

                                  sleep=30

                                   pmon=4

                                 quesiz=1

                                Reviver is ENABLED

                       

                      03-DEC-2018 11:34:35

                              Manager reconnected

                       

                      Searching process (30577)

                       

                      Connected process 3839093, session id 133861595, system pid 30577 after 0 tries

                       

                      AFPGMG--  03-DEC-2018 11:35:05

                              Current flags: AFPGMGMFAIL = FALSE, AFPGMGINCL = FALSE. Last flags: AFPGMGMFAIL = n/a, AFPGMGINCL = n/a

                       

                      AFPGMG--  03-DEC-2018 11:35:05

                              Manager process: spid=(16496), cpid=(3838945), ORA pid=(120) manager=(0/6269)

                              Process is a  SERVICE, continue to next manager

                       

                       

                      AFPGMG--  03-DEC-2018 11:35:05

                              Manager process: spid=(16515), cpid=(3838946), ORA pid=(123) manager=(0/6269)

                              Process is a  SERVICE, continue to next manager

                       

                       

                      AFPGMG--  03-DEC-2018 11:35:05

                              Manager process: spid=(16516), cpid=(3838947), ORA pid=(119) manager=(0/6269)

                              Process is a  SERVICE, continue to next manager

                       

                       

                      AFPGMG--  03-DEC-2018 11:35:05

                              Manager process: spid=(16525), cpid=(3838948), ORA pid=(122) manager=(0/6269)

                              Process is a  SERVICE, continue to next manager

                       

                       

                      AFPGMG--  03-DEC-2018 11:35:05

                              Manager process: spid=(16573), cpid=(3838952), ORA pid=(78) manager=(0/5)

                              Received lock, set mgralive=N

                       

                       

                              Manager process NOT reconnecting, mgralive=(N)03-DEC-2018 11:35:05

                              (3838952) Process_Status_Code = A

                              (3838952) Last TXN start is NULL

                              (3838952) Last TXN end is NULL

                      03-DEC-2018 11:35:05

                              (3838952) Exec RUNNING_REQUESTS cursor

                      03-DEC-2018 11:35:05

                              Manager process 3838952 not runnning any requests

                      sh: line 68: 30577 Segmentation fault      FNDLIBR FND CPMGR "FNDCPMBR sysmgr=\"\" sleep=30 pmon=4 quesiz=1 diag=y logfile=/u01/oracle/PPAPROD/fs_ne/inst/PPAPROD_oraapps-2/logs/appl/conc/log/PPAPROD_1203.mgr "  <<STOP

                      $unpw

                      STOP

                       

                      The PPAPROD_1203@PPAPROD internal concurrent manager has terminated with status 139 - giving up.

                      [applmgr@oraapps-2 log]$

                       

                       

                       

                      Please help....

                       

                      Kind regards,

                      • 8. Re: Concurrent Managers reset to zero?
                        mdtaylor

                        Please run cmclean.sql and see if it starts.  If it still fails, there is a Concurrent Manager Recovery wizard in Oracle Applications Manager Dashboard in Site Map, Diagnostics and Repair.

                        • 9. Re: Concurrent Managers reset to zero?
                          Beauty_and_dBest

                          Thanks Michael and all,

                           

                          I will try that,....brb

                          • 10. Re: Concurrent Managers reset to zero?
                            Kanda-Oracle

                            Hi JC,

                             

                            You are on 12.2.6. So you can run cpadmin which is designed to handle such issues (cmclean.sql in the past).

                             

                            EBS 12.2.5 :  New tool cpadmin.sh (CP Administration Main Menu)

                             

                            Thanks


                            N Kandasamy

                            • 11. Re: Concurrent Managers reset to zero?
                              Beauty_and_dBest

                              Hi Michael,

                               

                              cmclean.sql does magic!!!

                              It still worked in our 12.2.6

                               

                              The result is now successfull:

                               

                              Concurrent Manager's Name                  TARGET Processes ACTUAL Processes

                              ---------------------------------------------- ---------------- ----------------

                              C AQCART Service                                          0            0
                              Output Post Processor                                     4            4
                              Conflict Resolution Manager                               1            1
                              Internal Manager                                          1            1
                              Scheduler/Prereleaser Manager                             1            1
                              OAM Metrics Collection Manager                            1            1
                              Standard Manager                                        100          100
                              Workflow Agent Listener Service                           1            1
                              Workflow Mailer Service                                   1            1
                              Workflow Document Web Services Service                    1            1
                              PO Document Approval Manager                              1            1
                              Receiving Transaction Manager                             1            1
                              PA Streamline Manager                                     1            1
                              WMS Task Archiving Manager                                0            0
                              Inventory Manager                                         1            1
                              INV Remote Procedure Manager                              1            1
                              Marketing Data Mining Manager                             0            0
                              SFM Application Monitoring Service                        1            1
                              SFM Controller Service                                    0            0
                              SFM Event Manager Queue Service                           0            0
                              SFM Fulfillment Actions Queue Service                     0            0
                              SFM Fulfillment Element Ready Queue Service               0            0
                              SFM Inbound Messages Queue Service                        0            0
                              SFM Order Queue Service                                   0            0
                              SFM Timer Queue Service                                   0            0
                              SFM Work Item Queue Service                               0            0
                              SFM SM Interface Test Service                             0            0
                              Session History Cleanup                                   1            1
                              UWQ Worklist Items Release for Crashed session            1            1
                              Transportation Manager                                    1            1
                              FastFormula Transaction Manager                           0            0

                               

                              31 rows selected.

                               

                               

                               

                              Kind regards,

                              • 12. Re: Concurrent Managers reset to zero?
                                Beauty_and_dBest

                                Thanks Kanda,

                                 

                                I will try cpadmin to test if it will do magic as well

                                 

                                 

                                Kind regards,

                                 

                                 

                                PS.  why is that I can not tag "helpful" to my post? How do I convert it as question?

                                • 13. Re: Concurrent Managers reset to zero?
                                  Beauty_and_dBest

                                  Hi Michael, Kanda, and ALL.

                                   

                                  I tried login to the apps webpage but I see warning errors in the page,

                                  Although I can process the modules drill down successfully.

                                   

                                   

                                   

                                   

                                   

                                  .

                                   

                                  Kind regards,

                                  • 14. Re: Concurrent Managers reset to zero?
                                    hal9000akilroy

                                    cmclean is considered dangerous for new release

                                     

                                    abort any concurrent request for shutdown

                                    SELECT request_id,  status_code, phase_code

                                    FROM fnd_concurrent_requests

                                    WHERE program_application_id = 0

                                    AND concurrent_program_id = 5

                                    AND request_date > SYSDATE -1;

                                     

                                    and run manager cleanup (the new cmclean) in OAM

                                    1.png

                                    1 2 Previous Next