14 Replies Latest reply on Mar 27, 2014 5:32 PM by 984556

    Concurrent Manager starts but doesn't work

    984556

      Hello Everyone,

       

      Platform: AIX 5.3

      Oracle EBS 11.5.9

      DB 9i


      I have searched for an answer on this forum and on blogs but could not figure it out how to solve this issue.

       

      APPS tier, did a tar backup and restored on another machine ( everything is same ) but on the new machine concurrent manager doesn't run but everything else is running fine.

       

      While starting the APPS startup script:

       

      applprod@cloapp:/oraapp/prodcomn/admin/scripts/PROD_cloapp> apps/apps        <

       

       

      You are running adcmctl.sh version 115.19

       

       

      Starting concurrent manager for PROD ...

      Starting PROD_0323@PROD Internal Concurrent Manager

      Default printer is noprint

       

       

      adcmctl.sh: exiting with status 0

       

       

      applprod@cloapp:/oraapp/prodcomn/admin/scripts/PROD_cloapp>

       

      If i click on the internal log manager I get the following message:

       

      No log file exists for the process xxxxx. The concurrent manager may have encountered an error while attempting to create the log file

       

       

       

      Database Alert Log shows

       

      Starting up ORACLE RDBMS Version: 9.2.0.3.0.

      System parameters with non-default values:

        _trace_files_public      = TRUE

        processes                = 800

        sessions                 = 885

        timed_statistics         = TRUE

        shared_pool_size         = 301989888

        shared_pool_reserved_size= 30000000

        _shared_pool_reserved_min_alloc= 4100

        java_pool_size           = 67108864

        enqueue_resources        = 32000

        nls_language             = american

        nls_territory            = america

        nls_sort                 = binary

        nls_date_format          = DD-MON-RR

        nls_numeric_characters   = .,

        nls_comp                 = binary

        nls_length_semantics     = BYTE

        control_files            = /oradata/proddata/cntrl01.dbf, /oradata2/data/cntrl02.dbf, /oradata/proddata/cntrl03.dbf

        db_block_checksum        = TRUE

        db_block_size            = 8192

        db_cache_size            = 167772160

        compatible               = 9.2.0

        log_buffer               = 10485760

        log_checkpoint_interval  = 100000

        log_checkpoint_timeout   = 1200

        db_files                 = 512

        db_file_multiblock_read_count= 8

        log_checkpoints_to_alert = TRUE

        dml_locks                = 10000

        row_locking              = always

        undo_management          = AUTO

        undo_tablespace          = APPS_UNDOTS1

        undo_suppress_errors     = FALSE

        undo_retention           = 1800

        db_block_checking        = FALSE

        max_enabled_roles        = 100

        O7_DICTIONARY_ACCESSIBILITY= TRUE

        session_cached_cursors   = 200

        utl_file_dir             = /usr/tmp, /oratech/proddb/9.2.0/appsutil/outbound/PROD_clodb

        job_queue_processes      = 2

        _system_trig_enabled     = TRUE

        cursor_sharing           = EXACT

        parallel_min_servers     = 0

        parallel_max_servers     = 8

        background_dump_dest     = /oratech/proddb/9.2.0/admin/PROD_clodb/bdump

        user_dump_dest           = /oratech/proddb/9.2.0/admin/PROD_clodb/udump

        max_dump_file_size       = 20480

        core_dump_dest           = /oratech/proddb/9.2.0/admin/PROD_clodb/cdump

        optimizer_features_enable= 9.2.0

        db_name                  = PROD

        open_cursors             = 600

        ifile                    = /oratech/proddb/9.2.0/dbs/PROD_clodb_ifile.ora

        sql_trace                = FALSE

        _sort_elimination_cost_ratio= 5

        _b_tree_bitmap_plans     = FALSE

        _fast_full_scan_enabled  = FALSE

        optimizer_max_permutations= 2000

        query_rewrite_enabled    = true

        _index_join_enabled      = FALSE

        _sqlexec_progression_cost= 2147483647

        _like_with_bind_as_equality= TRUE

        pga_aggregate_target     = 1073741824

        workarea_size_policy     = AUTO

        aq_tm_processes          = 1

        olap_page_pool_size      = 4194304

      PMON started with pid=2

      DBW0 started with pid=3

      LGWR started with pid=4

      CKPT started with pid=5

      SMON started with pid=6

      RECO started with pid=7

      CJQ0 started with pid=8

      QMN0 started with pid=9

      Sun Mar 23 18:10:09 2014

      ALTER DATABASE   MOUNT

      Sun Mar 23 18:10:13 2014

      Successful mount of redo thread 1, with mount id 254776577.

      Sun Mar 23 18:10:13 2014

      Database mounted in Exclusive Mode.

      Completed: ALTER DATABASE   MOUNT

      Sun Mar 23 18:10:13 2014

      ALTER DATABASE OPEN

      Sun Mar 23 18:10:15 2014

      Thread 1 opened at log sequence 9194

        Current log# 2 seq# 9194 mem# 0: /oradata/proddata/log02a.dbf

        Current log# 2 seq# 9194 mem# 1: /orada

       

       

      Where should I start looking inorder to resolve this issue.

        • 1. Re: Concurrent Manager starts but doesn't work
          Hussein Sawwan-Oracle

          Please do CM recovery as per (Concurrent Processing - Concurrent Manager Recovery Troubleshooting Wizard (Doc ID 134007.1)) and check then -- Post the contents of the CM log file.

           

          You may also run (Concurrent Processing - CCM.sql Diagnostic Script to Diagnose Common Concurrent Manager Issues (Doc ID 171855.1)).

           

          Thanks,

          Hussein

          1 person found this helpful
          • 2. Re: Concurrent Manager starts but doesn't work
            984556

            I read both metalinks and followed the instructions, and this is what I did

             

            1) Stopped Apps

             

            2) on the DB Tier ,  Ran ccmclean.sql

             

             

             

            oraprod@clodb:/oratech/proddb/9.2.0>sqlplus apps/apps @cmclean.sql

             

             

            SQL*Plus: Release 9.2.0.3.0 - Production on Mon Mar 24 15:08:27 2014

             

             

            Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

             

             

             

             

            Connected to:

            Oracle9i Enterprise Edition Release 9.2.0.3.0 - 64bit Production

            With the Partitioning, OLAP and Oracle Data Mining options

            JServer Release 9.2.0.3.0 - Production

             

             

            DOC>

            DOC>   WARNING : Do not run this script without explicit instructions

            DOC>             from Oracle Support

            DOC>

            DOC>

            DOC>   *** Make sure that the managers are shut down     ***

            DOC>   *** before running this script                    ***

            DOC>

            DOC>   *** If the concurrent managers are NOT shut down, ***

            DOC>   *** exit this script now !!                       ***

            DOC>

            DOC>#

            If you wish to continue type the word 'dual': dual

             

             

             

             

             

             

            -----------------------------------------------------------------------

            -- Updating invalid process status codes in FND_CONCURRENT_PROCESSES

             

             

            0 rows updated.

             

             

             

             

            -----------------------------------------------------------------------

            -- Updating running processes in FND_CONCURRENT_QUEUES

            -- Setting running_processes = 0 and max_processes = 0 for all managers

             

             

            36 rows updated.

             

             

             

             

            -----------------------------------------------------------------------

            -- Updating invalid control_codes in FND_CONCURRENT_QUEUES

             

             

            0 rows updated.

             

             

             

             

            36 rows updated.

             

             

             

             

            -----------------------------------------------------------------------

            -- Updating any Running or Terminating requests to Completed/Error

             

             

            0 rows updated.

             

             

             

             

            -----------------------------------------------------------------------

            -- Updating any Runalone flags to 'N'

             

             

            Updated 0 rows of runalone_flag in fnd_conflicts_domain to 'N'

             

             

            -----------------------------------------------------------------------

            Updates complete.

            Type commit now to commit these updates, or rollback to cancel.

            -----------------------------------------------------------------------

             

             

            >

            > commit

             

             

            SQL>

             

            After this I ran  sqlplus apps/apps @ccm.sql

             

            >sqlplus apps/apps @ccm.sql

            SQL*Plus: Release 9.2.0.3.0 - Production on Mon Mar 24 15:13:17 2014

             

             

            Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

             

             

             

             

            Connected to:

            Oracle9i Enterprise Edition Release 9.2.0.3.0 - 64bit Production

            With the Partitioning, OLAP and Oracle Data Mining options

            JServer Release 9.2.0.3.0 - Production

             

             

             

             

            Step 1 Checking how many rows in FND_CONCURRENT_REQUEST.

             

             

              COUNT(*)

            ----------

                   954

             

             

             

             

             

             

            Step 2 Checking how many rows in FND_CONCURRENT_PROCESSES table.

             

             

              COUNT(*)

            ----------

                 68571

             

             

             

             

             

             

            Step 3 Checking sys.dual table which should have one and only one row.

             

             

              COUNT(*)

            ----------

                     1

             

             

            If you have more than one row in sys.dual, please delete it

            sql> delete rownum from SYS.DUAL

            rownum= the row number to delete

             

             

             

             

             

             

             

            Step 4 Checking fnd_dual. There must be at lest one row:

             

             

              COUNT(*)

            ----------

                     1

             

             

            If there are no record selected,

            Update fnd_dual table to have at lest one record

             

             

             

             

            Step 5 Checking the Internal Manager queue name "FNDICM" which should be=1

             

             

            CONCURRENT_QUEUE_ID

            -------------------

                              1

             

             

             

             

             

            Step 6 Checking for Active processes under the Internal Manager queue

            in fnd_concurrent_proceses table:

             

             

             

             

            no rows selected

             

             

            If any rows found with process_status_code with value = 'A' (= Active)

            The internal Manager will not start up ,so to avoide this issue

            update these rows to have process_status_code value ='K'(terminated)

             

             

             

             

             

            Step 7 Checking for Active processes under the Standard Manager queue

            in fnd_concurrent_proceses table:

             

             

             

             

            no rows selected

             

             

            If any rows found with process_status_code with value = 'A' (= Active)

            The internal Manager will not start up ,so to avoide this issue

            update these rows to have process_status_code value ='K'(terminated)

             

             

             

             

             

            Step 8 Checking for Active processes under the Conflict Manager queue

            in fnd_concurrent_proceses table:

             

             

             

             

            no rows selected

             

             

            If any rows found with process_status_code with value = 'A' (= Active)

            The internal Manager will not start up ,so to avoide this issue

            update these rows to have process_status_code value ='K'(terminated)

             

             

             

             

            Step 9 Checking Actual and Target Processes for Internal Manager:

             

             

            MAX_PROCESSES RUNNING_PROCESSES

            ------------- -----------------

                        0                 0

             

             

            If the MAX_PROCESSES=RUNNING_PROCESSES that means the manager is UP.

             

             

             

             

            Step 10 Checking Actual and Target Processes for the Standard Manager:

             

             

            MAX_PROCESSES RUNNING_PROCESSES

            ------------- -----------------

                        0                 0

             

             

            If the MAX_PROCESSES=RUNNING_PROCESSES that means the manager is UP.

             

             

             

             

            Step 11 Checking Actual and Target Processes for Conflict Resolution  Manager:

             

             

            MAX_PROCESSES RUNNING_PROCESSES

            ------------- -----------------

                        0                 0

             

             

            If the MAX_PROCESSES=RUNNING_PROCESSES that means the manager is UP.

             

             

             

             

            Step 12 Checking if the control_code set to 'N':

             

             

            no rows selected

             

             

             

             

            If any rows selected, please update the table fnd_concurrent_queues:

            Update fnd_concurrent_queues set control_code = null

            where control_code ='N'

            Update fnd_concurrent_queues set target_node = null

            commit

             

             

             

             

            Step 13 Checking terminated processes:

             

             

             

             

              COUNT(*)

            ----------

                     0

             

             

             

             

            If you have terminated processes run the following sql statement:

             

             

            SQL> Update fnd_concurrent_requests

            set status_code = 'E', phase_code = 'C'

            where status_code = 'T'

             

             

            Step 14 Checking pending requests:

             

             

              COUNT(*)

            ----------

                     0

             

             

            If any rows selected please run the following sql statement:

            SQL> Update fnd_concurrent_requests

            set status_code = 'E', phase_code = 'C'

            where status_code = 'P'

             

             

            Step 15 Checking Running processes:

             

             

             

             

              COUNT(*)

            ----------

                     0

             

             

             

             

            If you have Running processes run the following sql statement

            SQL> Update fnd_concurrent_requests

            set status_code = 'E', phase_code = 'C'

            where status_code = 'R'

             

             

             

            Step 16 Checking the PMON method, which should be set to LOCK:

             

             

             

             

            no rows selected

             

             

             

             

            If the PROFILE_OPTION_VALUE was't LOCK please

            Reset PMON to LOCK by running afimpmon.sql script(The manager should be down)

            1-At UNIX command prompt:

            2-cd $FND_TOP/sql

            3-Log into SQLPLUS as apps/

            SQL> @afimpmon.sql

            prompt1:dual

            prompt2:LOCK     (LOCK MUST BE ALL UPPERCASE)

            For Oracle Applications Release 11.5 and 12.0, when you check the PMON

            Method you may get no rows selected which is normal,

            because in apps 11.5 and 12.0 the PMON Method is hard coded to Lock at

            the Operating System level.

             

             

             

             

             

            Step-17 Checking how many FNDLIBR processes are running:

            -For Unix :From unix command  prompt $ ps -ef |grep -i fndlibr

            If you have any FNDLIBR processes running,please kill them before

            starting or shuting down the internal manager

             

             

             

             

            -For NT, through Task Manager, check the entries under the Processes tab

            SP2-0734: unknown command beginning "for FNDLIB..." - rest of line ignored.

            If there are any, Highlight and click [End Process] button to kill processes

             

             

            Step-18 Checking how many "FND_%"invalid objects:

             

             

            no rows selected

             

             

            If you have any invalied objects please see Note 113947.1

             

             

             

            Step-19-How to find the PID in the O/S for request_id:

            If you do not like to check this enter any number then click Enter to Exit

            Enter value for request_id:

             

             

             

             

             

            Searching for CCM log files

             

            applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>ls -la *.log | grep 'Mar 24'    

            -rw-r--r--    1 applprod dba            5900 Mar 24 11:06 03241106.log

            -rw-r--r--    1 applprod dba            6033 Mar 24 11:22 03241121.log

            -rw-r--r--    1 applprod dba            5980 Mar 24 12:03 03241203.log

            -rw-r--r--    1 applprod dba            5701 Mar 24 11:10 Errors18.log

            -rw-r--r--    1 applprod dba             703 Mar 24 11:22 Errors19.log

            -rw-r--r--    1 applprod dba            5172 Mar 24 11:06 Events18.log

            -rw-r--r--    1 applprod dba            6155 Mar 24 11:22 Events19.log

            -rw-r--r--    1 applprod dba            6820 Mar 24 11:22 system.log

            applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>

             

             

            >Errors19.log          <

            Start Time --- --Mon Mar 24 11:22:16 AST 2014--less than 1 millisecond--

             

             

            -- <1> --

            Time       : --Mon Mar 24 11:22:25 AST 2014--8 seconds, 899 milliseconds--

            Note       : PORT CONFLICT '9300'

            java.net.BindException: The socket name is already in use.

                    at java.net.PlainDatagramSocketImpl.bind(Native Method)

                    at java.net.DatagramSocket.create(DatagramSocket.java:188)

                    at java.net.DatagramSocket.<init>(DatagramSocket.java:160)

                    at java.net.DatagramSocket.<init>(DatagramSocket.java:132)

                    at oracle.apps.jtf.fm.engine.processor.RemoteMonitor.openSocket(RemoteMonitor.java:309)

                    at oracle.apps.jtf.fm.engine.processor.RemoteMonitor.run(RemoteMonitor.java:155)

                    at java.lang.Thread.run(Thread.java:513)

             

             

             

            Setting service OracleConcMgrPROD_cloapp to mode 3

            Executing service control script:

            /oraapp/prodcomn/admin/scripts/PROD_cloapp/adcmctl.sh stop

            Timeout specified in context file: 1000 second(s)

             

             

            script returned:

            ****************************************************

             

             

            You are running adcmctl.sh version 115.19

             

             

            Shutting down concurrent managers for PROD ...

            Submitted request 2560347 for CONCURRENT FND SHUTDOWN

             

             

             

             

            adcmctl.sh: exiting with status 0

             

             

             

             

            .end std out.

             

             

            .end err out.

             

             

            ****************************************************

             

             

             

             

             

             

            [Service Control Execution Report]

            The report format is:

              <Service>                                    <Script>         <Status>

             

             

              Oracle Apache Server PROD_cloapp             adapcctl.sh      Stopped

              OracleTNSListener80APPS_PROD_cloapp          adalnctl.sh      Stopped

              OracleConcMgrPROD_cloapp                     adcmctl.sh       Stopped

              OracleFormsServer-Forms60PROD_cloapp         adfrmctl.sh      Stopped

              OracleReportServer-Rep60_PROD                adrepctl.sh      Stopped

              Oracle Metrics Client PROD_cloapp            adfmcctl.sh      Stopped

              Oracle Metrics Server PROD_cloapp            adfmsctl.sh      Stopped

              Oracle Fulfillment Server PROD_cloapp        jtffmctl.sh      Stopped

              Oracle Discoverer services PROD_cloapp       addisctl.sh      Stopped

              Oracle Restricted Apache Server PROD_cloapp  adaprstctl.sh    Disabled

              Oracle Apache Server PROD_cloapp for PL/SQL  adapcctl.sh      Disabled

              Oracle TCF SocketServer PROD_cloapp          adtcfctl.sh      Disabled

              Oracle ICSM PROD_cloapp                      ieoicsm.sh       Disabled

              Oracle iProcurement Bulk Loader PROD_cloapp  icxblkctl.sh     Disabled

             

             

             

             

            ServiceControl is exiting with status 0

             

             

            applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>

            applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>

             

             

             

             

            Can't really figure it out how to resolve this issue.

            • 3. Re: Concurrent Manager starts but doesn't work
              Shakti1807

              hi,

              Please check if the the file system is mounted correctly, if df -k returns correct value without any error. It is possible that concurrent managers are not able to create log/out file in your new server/file system and hence not working.

               

              Thanks

              • 4. Re: Concurrent Manager starts but doesn't work
                Hussein Sawwan-Oracle

                Please stop all the services then verify that all OS processes are killed (or reboot the box) and try again.

                 

                Thanks,

                Hussein

                • 5. Re: Concurrent Manager starts but doesn't work
                  984556

                  @Shakti1803

                  DB server

                  root@clodb:/>df -k

                  Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on

                  /dev/hd4          1572864   1350044   15%     2259     1% /

                  /dev/hd2          3145728   1031832   68%    34083    13% /usr

                  /dev/hd9var       1048576    702036   34%     4623     3% /var

                  /dev/hd3          1048576   1042932    1%       59     1% /tmp

                  /dev/hd1          1048576    807740   23%       76     1% /home

                  /proc                   -         -    -         -     -  /proc

                  /dev/hd10opt       524288    368664   30%     3710     5% /opt

                  /dev/livedump      524288    523880    1%        4     1% /var/adm/ras/livedump

                  /dev/fslv03      19922944  17673368   12%        6     1% /backup

                  /dev/fslv02     146800640  14883040   90%      421     1% /oradata

                  /dev/fslv05      31457280   5543968   83%       36     1% /oradata2

                  /dev/fslv06      20971520  12271512   42%   133390     5% /oratech

                   

                   

                  Apps Server

                  root@cloapp:/>df -k

                  Filesystem    1024-blocks      Free %Used    Iused %Iused Mounted on

                  /dev/hd4          1048576    862100   18%     1734     1% /

                  /dev/hd2          2621440    981424   63%    29096    12% /usr

                  /dev/hd9var       1048576    672576   36%     1534     2% /var

                  /dev/hd3          1572864   1553620    2%      158     1% /tmp

                  /dev/hd1          1048576    805272   24%      155     1% /home

                  /proc                   -         -    -         -     -  /proc

                  /dev/hd10opt       524288    416192   21%     2353     3% /opt

                  /dev/fslv01      29360128   7597068   75%        6     1% /ebs1

                  /dev/fslv00      20971520  10950668   48%    21042     1% /backup

                  /dev/fslv02      71303168   3516132   96%   710887    48% /oraapp

                  root@cloapp:/>

                   

                  @Hussein

                   

                  1. Rebooted both DB and Apps Server

                   

                  Starting Database

                   

                  Starting ORACLE instance (normal)

                  LICENSE_MAX_SESSION = 0

                  LICENSE_SESSIONS_WARNING = 0

                  SCN scheme 3

                  Using log_archive_dest parameter default value

                  LICENSE_MAX_USERS = 0

                  SYS auditing is disabled

                  Starting up ORACLE RDBMS Version: 9.2.0.3.0.

                  System parameters with non-default values:

                    _trace_files_public      = TRUE

                    processes                = 800

                    sessions                 = 885

                    timed_statistics         = TRUE

                    shared_pool_size         = 301989888

                    shared_pool_reserved_size= 30000000

                    _shared_pool_reserved_min_alloc= 4100

                    java_pool_size           = 67108864

                    enqueue_resources        = 32000

                    nls_language             = american

                    nls_territory            = america

                    nls_sort                 = binary

                    nls_date_format          = DD-MON-RR

                    nls_numeric_characters   = .,

                    nls_comp                 = binary

                    nls_length_semantics     = BYTE

                    control_files            = /oradata/proddata/cntrl01.dbf, /oradata2/data/cntrl02.dbf, /oradata/proddata/cntrl03.dbf

                    db_block_checksum        = TRUE

                    db_block_size            = 8192

                    db_cache_size            = 167772160

                    compatible               = 9.2.0

                    log_buffer               = 10485760

                    log_checkpoint_interval  = 100000

                    log_checkpoint_timeout   = 1200

                    db_files                 = 512

                    db_file_multiblock_read_count= 8

                    log_checkpoints_to_alert = TRUE

                    dml_locks                = 10000

                    row_locking              = always

                    undo_management          = AUTO

                    undo_tablespace          = APPS_UNDOTS1

                    undo_suppress_errors     = FALSE

                    undo_retention           = 1800

                    db_block_checking        = FALSE

                    max_enabled_roles        = 100

                    O7_DICTIONARY_ACCESSIBILITY= TRUE

                    session_cached_cursors   = 200

                    utl_file_dir             = /usr/tmp, /oratech/proddb/9.2.0/appsutil/outbound/PROD_clodb

                    job_queue_processes      = 2

                    _system_trig_enabled     = TRUE

                    cursor_sharing           = EXACT

                    parallel_min_servers     = 0

                    parallel_max_servers     = 8

                    background_dump_dest     = /oratech/proddb/9.2.0/admin/PROD_clodb/bdump

                    user_dump_dest           = /oratech/proddb/9.2.0/admin/PROD_clodb/udump

                    max_dump_file_size       = 20480

                    core_dump_dest           = /oratech/proddb/9.2.0/admin/PROD_clodb/cdump

                    optimizer_features_enable= 9.2.0

                    db_name                  = PROD

                    open_cursors             = 600

                    ifile                    = /oratech/proddb/9.2.0/dbs/PROD_clodb_ifile.ora

                    sql_trace                = FALSE

                    _sort_elimination_cost_ratio= 5

                    _b_tree_bitmap_plans     = FALSE

                    _fast_full_scan_enabled  = FALSE

                    optimizer_max_permutations= 2000

                    query_rewrite_enabled    = true

                    _index_join_enabled      = FALSE

                    _sqlexec_progression_cost= 2147483647

                    _like_with_bind_as_equality= TRUE

                    pga_aggregate_target     = 1073741824

                    workarea_size_policy     = AUTO

                    aq_tm_processes          = 1

                    olap_page_pool_size      = 4194304

                  PMON started with pid=2

                  DBW0 started with pid=3

                  LGWR started with pid=4

                  CKPT started with pid=5

                  SMON started with pid=6

                  RECO started with pid=7

                  CJQ0 started with pid=8

                  QMN0 started with pid=9

                  Wed Mar 26 18:06:49 2014

                  ALTER DATABASE   MOUNT

                  Wed Mar 26 18:06:54 2014

                  Successful mount of redo thread 1, with mount id 255049657.

                  Wed Mar 26 18:06:54 2014

                  Database mounted in Exclusive Mode.

                  Completed: ALTER DATABASE   MOUNT

                  Wed Mar 26 18:06:54 2014

                  ALTER DATABASE OPEN

                  Wed Mar 26 18:06:55 2014

                  Beginning crash recovery of 1 threads

                  Wed Mar 26 18:06:56 2014

                  Started first pass scan

                  Wed Mar 26 18:06:56 2014

                  Completed first pass scan

                  6675 redo blocks read, 122 data blocks need recovery

                  Wed Mar 26 18:06:56 2014

                  Started recovery at

                  Thread 1: logseq 9245, block 15592, scn 0.0

                  Recovery of Online Redo Log: Thread 1 Group 2 Seq 9245 Reading mem 0

                    Mem# 0 errs 0: /oradata/proddata/log02a.dbf

                    Mem# 1 errs 0: /oradata/proddata/log02b.dbf

                  Recovery of Online Redo Log: Thread 1 Group 1 Seq 9246 Reading mem 0

                    Mem# 0 errs 0: /oradata/proddata/log01a.dbf

                    Mem# 1 errs 0: /oradata/proddata/log01b.dbf

                  Wed Mar 26 18:06:56 2014

                  Completed redo application

                  Wed Mar 26 18:06:56 2014

                  Ended recovery at

                  Thread 1: logseq 9246, block 1792, scn 0.3903148237

                  122 data blocks read, 122 data blocks written, 6675 redo blocks read

                  Crash recovery completed successfully

                  Wed Mar 26 18:06:57 2014

                  Thread 1 advanced to log sequence 9247

                  Thread 1 opened at log sequence 9247

                    Current log# 3 seq# 9247 mem# 0: /oradata/proddata/log03a.dbf

                    Current log# 3 seq# 9247 mem# 1: /oradata/proddata/log03b.dbf

                  Successful open of redo thread 1.

                  Wed Mar 26 18:06:57 2014

                  MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set

                  Wed Mar 26 18:06:57 2014

                  SMON: enabling cache recovery

                  Wed Mar 26 18:07:00 2014

                  Undo Segment 1 Onlined

                  Undo Segment 2 Onlined

                  Undo Segment 3 Onlined

                  Undo Segment 4 Onlined

                  Undo Segment 5 Onlined

                  Undo Segment 6 Onlined

                  Undo Segment 7 Onlined

                  Undo Segment 8 Onlined

                  Undo Segment 9 Onlined

                  Undo Segment 10 Onlined

                  Successfully onlined Undo Tablespace 371.

                  Wed Mar 26 18:07:00 2014

                  SMON: enabling tx recovery

                  Wed Mar 26 18:07:00 2014

                  Database Characterset is AR8ISO8859P6

                  replication_dependency_tracking turned off (no async multimaster replication found)

                  Completed: ALTER DATABASE OPEN

                   

                   

                  2. Starting Apps

                   

                  root@cloapp:/>su - applprod

                  [YOU HAVE NEW MAIL]

                  applprod@cloapp:/home/applprod>./startup.sh

                   

                   

                  You are running adstrtal.sh version 115.14

                   

                   

                  Executing service control script:

                  /oraapp/prodcomn/admin/scripts/PROD_cloapp/adapcctl.sh start

                  script returned:

                  ****************************************************

                   

                   

                  adapcctl.sh version 115.43

                   

                   

                  Starting Apache Web Server Listener (dedicated HTTP) ...

                  Starting Apache Web Server Listener (dedicated PLSQL) ...

                   

                   

                  adapcctl.sh: exiting with status 0

                   

                   

                   

                   

                  .end std out.

                   

                   

                  .end err out.

                   

                   

                  ****************************************************

                   

                   

                   

                   

                  Executing service control script:

                  /oraapp/prodcomn/admin/scripts/PROD_cloapp/adalnctl.sh start

                  script returned:

                  ****************************************************

                   

                   

                  adalnctl.sh version

                   

                   

                  Checking for FNDFS executable.

                  Starting listener process APPS_PROD.

                   

                   

                  adalnctl.sh: exiting with status 0

                   

                   

                   

                   

                  .end std out.

                   

                   

                  .end err out.

                   

                   

                  ****************************************************

                   

                   

                   

                   

                  Executing service control script:

                  /oraapp/prodcomn/admin/scripts/PROD_cloapp/adcmctl.sh start

                  script returned:

                  ****************************************************

                   

                   

                  You are running adcmctl.sh version 115.19

                   

                   

                  Starting concurrent manager for PROD ...

                  Starting PROD_0326@PROD Internal Concurrent Manager

                  Default printer is noprint

                   

                   

                  adcmctl.sh: exiting with status 0

                   

                   

                   

                   

                  .end std out.

                   

                   

                  .end err out.

                   

                   

                  ****************************************************

                   

                   

                   

                   

                  Executing service control script:

                  /oraapp/prodcomn/admin/scripts/PROD_cloapp/adfrmctl.sh start

                  script returned:

                  ****************************************************

                   

                   

                  You are running adfrmctl.sh version 115.33

                   

                   

                  Starting forms server for PROD on port 9000.

                   

                   

                  adfrmctl.sh: exiting with status 0

                   

                   

                   

                   

                  .end std out.

                   

                   

                  .end err out.

                   

                   

                  ****************************************************

                   

                   

                   

                   

                  Executing service control script:

                  /oraapp/prodcomn/admin/scripts/PROD_cloapp/adrepctl.sh start

                  script returned:

                  ****************************************************

                   

                   

                  You are running adrepctl.sh version 115.29

                   

                   

                  starting Reports Server for PROD on port 7000.

                   

                   

                  adrepctl.sh: exiting with status 0

                   

                   

                   

                   

                  .end std out.

                   

                   

                  .end err out.

                   

                   

                  ****************************************************

                   

                   

                   

                   

                  Executing service control script:

                  /oraapp/prodcomn/admin/scripts/PROD_cloapp/adfmcctl.sh start

                  script returned:

                  ****************************************************

                   

                   

                  You are running adfmcctl.sh version 115.16

                   

                   

                  Starting forms load balancing client for PROD.

                   

                   

                  adfmcctl.sh: exiting with status 0

                   

                   

                   

                   

                  .end std out.

                   

                   

                  .end err out.

                   

                   

                  ****************************************************

                   

                   

                   

                   

                  Executing service control script:

                  /oraapp/prodcomn/admin/scripts/PROD_cloapp/adfmsctl.sh start

                  script returned:

                  ****************************************************

                   

                   

                  You are running adfmsctl.sh version 115.12

                   

                   

                  starting forms  metrics server for PROD.

                   

                   

                  adfmsctl.sh: exiting with status 0

                   

                   

                   

                   

                  .end std out.

                   

                   

                  .end err out.

                   

                   

                  ****************************************************

                   

                   

                   

                   

                  Executing service control script:

                  /oraapp/prodcomn/admin/scripts/PROD_cloapp/jtffmctl.sh start

                  script returned:

                  ****************************************************

                   

                   

                  You are running jtffmctl.sh version 115.7

                   

                   

                  Starting Fulfillment Server for PROD on port 9300 ...

                   

                   

                  jtffmctl.sh: exiting with status 0

                   

                   

                   

                   

                  .end std out.

                   

                   

                  .end err out.

                   

                   

                  ****************************************************

                   

                   

                   

                   

                  Executing service control script:

                  /oraapp/prodcomn/admin/scripts/PROD_cloapp/addisctl.sh start

                  script returned:

                  ****************************************************

                   

                   

                  addisctl.sh version 115.8

                   

                   

                  Oracle Discoverer OAD is not running (no process with PID 426196 )

                   

                   

                  Waiting for OAD to start...

                  Started OAD.

                  OAD logs messages to the file /oraapp/prodora/8.0.6/discwb4/util/oad.log.

                  Oracle Discoverer OSAGENT is not running (no process with PID 442586)

                   

                   

                  /oraapp/prodora/8.0.6/vbroker/bin/osagent

                  Started osagent.

                  Osagent logs messages to the file /oraapp/prodora/8.0.6/discwb4/util/osagent.log.

                  Oracle Discoverer LOCATOR is not running (no process with PID 139482)

                   

                   

                  Discoverer Locator Started.

                  Locator logs messages to the file /oraapp/prodora/8.0.6/discwb4/util/locator.log.

                   

                   

                  addisctl.sh: exiting with status 0

                   

                   

                   

                   

                  .end std out.

                   

                   

                  .end err out.

                   

                   

                  ****************************************************

                   

                   

                   

                   

                  All enabled services for this node are started.

                  Check logfile /oraapp/prodcomn/admin/log/PROD_cloapp/03261508.log for details

                   

                   

                  Exiting with status 0

                  applprod@cloapp:/home/applprod>

                   

                   

                  3. Going to the log directory of Concurrent Manager in the Apps Server

                  applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>ls -la *.mgr | grep 26

                  -rw-r--r--   1 applprod dba            2627 Mar 26 15:10 PROD_0326.mgr

                  -rw-r--r--   1 applprod dba         1269795 Jan 05 12:37 PROD_1230.mgr

                  -rw-r--r--   1 applprod dba          258264 Dec 29 22:27 w15097.mgr

                  -rw-r--r--   1 applprod dba          268420 Dec 29 22:27 w15098.mgr

                  applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>cat PROD_0326.mgr

                   

                   

                  ========================================================================

                  Starting PROD_0326@PROD Internal Concurrent Manager -- shell process ID 422094

                   

                   

                            logfile=/oraapp/prodcomn/admin/log/PROD_cloapp/PROD_0326.mgr

                            PRINTER=noprint

                             mailto=applprod

                            restart=N

                               diag=N

                              sleep=60 (default)

                               pmon=20 (default)

                             quesiz=1  (default)

                   

                   

                  establish_icm - Unable to insert ICM record

                  Call to establish_icm failed

                  The Internal Concurrent Manager has encountered an error.

                   

                   

                  Review concurrent manager log file for more detailed information. : 26-MAR-2014                   15:10:12 -

                   

                   

                   

                   

                  Shutting down Internal Concurrent Manager : 26-MAR-2014 15:10:12

                   

                   

                  List of errors encountered:

                  .............................................................................

                   

                   

                  _ 1 _

                  Routine AFPCSQ encountered an ORACLE error.  .

                   

                   

                  Review your error messages for the cause of the error. (=<POINTER>)

                  .............................................................................

                   

                   

                   

                   

                  APP-FND-01564: ORACLE error 1653 in insert_icm_record

                   

                   

                  Cause:  insert_icm_record failed due to ORA-01653: unable to extend table APPLSY                  S.FND_CONCURRENT_PROCESSES by 65 in tablespace APPLSYSD.

                   

                   

                  The SQL statement being executed at the time of the error was: INSERT   INTO FND                  _CONCURRENT_PROCESSES       (CONCURRENT_PROCESS_ID, LAST_UPDATE_DATE,        CRE                  ATION_DATE,         CREATED_BY,        LAST_UPDATED_BY,       ORACLE_PROCESS_ID,                          QUEUE_APPLICATION_ID,  CONCURRENT_QUEUE_ID,        OS_PROCESS_ID,                           SESSION_ID,        PROCESS_START_DATE,    PROCESS_STATUS_CODE,        MANAGER_                  TYPE,          NODE_NAME,        DB_NAME,               DB_DOMAIN,        DB_INS                  TANCE, Instance_Number) VALUES       (:cpid,                 SYSDATE,        SYS                  DATE,               :mgrusrid,        :mgrusrid,             :opid,        :qapp                  id,               :qid,        :ospid,                :osid,        SYSDATE,                                 'A',        0,                     :node,        :dbname,                                 :dbdomain,        :dbinst, (Select instance_number from v$instance)) and was                   executed from the file &ERRFILE.

                   

                   

                  List of errors encountered:

                  .............................................................................

                   

                   

                  _ 1 _

                  Routine AFPCAL received failure code while parsing or running your

                  concurrent program CPMGR

                   

                   

                  Review your concurrent request log file for more detailed information.

                  Make sure you are passing arguments in the correct format.

                  .............................................................................

                   

                   

                  The PROD_0326@PROD internal concurrent manager has terminated with status 1 - gi                  ving up.

                  applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>

                   

                  -rw-r--r--   1 applprod dba            2026 Jan 09 11:31 w58488.mgr

                  -rw-r--r--   1 applprod dba            2608 Jan 15 11:15 w61305.mgr

                  -rw-r--r--   1 applprod dba           26932 Jan 15 11:15 w61316.mgr

                  -rw-r--r--   1 applprod dba            2609 Mar 22 18:44 w85610.mgr

                   

                  what does this mean ? thanks

                  • 6. Re: Concurrent Manager starts but doesn't work
                    Shakti1807

                    Hi ,

                    It seems that the tablespace APPLSYSD is full.You need to add some space in this tablespace by resizing the datafiles of this tablespace or by adding datafile. As there is not enough space in tablespace, manager process are not able to insert any new row in FND_CONCURRENT_PROCESSES table.

                     

                    Thanks,

                    Shakti

                    • 7. Re: Concurrent Manager starts but doesn't work
                      984556

                      Thank you, however,

                      ALTER TABLESPACE APPLSYSD ADD DATAFILE '/oratech/applsysd.dbf' SIZE 1024M AUTOEXTEND OFF

                      Wed Mar 26 22:05:15 2014

                      Completed: ALTER TABLESPACE APPLSYSD ADD DATAFILE '/oratech/a

                       

                      Stopped Apps and Stopped Database

                       

                       

                      something went abnormal while shutting down APPS

                       

                      applprod@cloapp:/home/applprod>./stop.sh

                       

                       

                      You are running adstpall.sh version 115.17

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adapcctl.sh stop

                      script returned:

                      ****************************************************

                       

                       

                      adapcctl.sh version 115.43

                       

                       

                      Stopping Apache Web Server Listener (dedicated HTTP) ...

                      Stopping Apache Web Server Listener (dedicated PLSQL) ...

                      /oraapp/prodora/iAS/Apache/Apache/bin/apachectl stop: httpd stopped

                       

                       

                      adapcctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adalnctl.sh stop

                      script returned:

                      ****************************************************

                       

                       

                      adalnctl.sh version

                       

                       

                      Shutting down listener process APPS_PROD.

                       

                       

                      adalnctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adcmctl.sh stop

                      script returned:

                      ****************************************************

                       

                       

                      You are running adcmctl.sh version 115.19

                       

                       

                      Shutting down concurrent managers for PROD ...

                      Submitted request 2560348 for CONCURRENT FND SHUTDOWN

                       

                       

                       

                       

                      adcmctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adfrmctl.sh stop

                      script returned:

                      ****************************************************

                       

                       

                      You are running adfrmctl.sh version 115.33

                       

                       

                      Stopping forms server for PROD on port 9000.

                       

                       

                      adfrmctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adrepctl.sh stop

                      script returned:

                      ****************************************************

                       

                       

                      You are running adrepctl.sh version 115.29

                       

                       

                      stopping Reports Server for PROD on port 7000.

                      kill -9 426196

                      Reports Server for PROD successfully stopped.

                       

                       

                      adrepctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                      Usage: ipcrm [-q msqid] [-m shmid] [-s semid]

                             [-Q msgkey] [-M shmkey] [-S semkey].

                      Usage: ipcrm -r {-q|-m|-s} name

                      Usage: ipcrm -r -u [-o owner] [-g group]

                       

                       

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adfmcctl.sh stop

                      script returned:

                      ****************************************************

                       

                       

                      You are running adfmcctl.sh version 115.16

                       

                       

                      Stopping forms load balancing client for PROD.

                       

                       

                      adfmcctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adfmsctl.sh stop

                      script returned:

                      ****************************************************

                       

                       

                      You are running adfmsctl.sh version 115.12

                       

                       

                      Stopping forms metrics server for PROD on port 9200.

                       

                       

                      adfmsctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/jtffmctl.sh stop

                      script returned:

                      ****************************************************

                       

                       

                      You are running jtffmctl.sh version 115.7

                       

                       

                      Shutting down Fulfillment Server for PROD on port 9300 ...

                       

                       

                      jtffmctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/addisctl.sh stop

                      script returned:

                      ****************************************************

                       

                       

                      addisctl.sh version 115.8

                       

                       

                      OAD is stopped

                      Osagent is stopped

                      Locator is stopped

                      Unable to stop gatekeeper. No process-id  file found.

                      other discoverer processes stopped

                       

                       

                      addisctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      All enabled services on this node are stopped.

                      Check logfile /oraapp/prodcomn/admin/log/PROD_cloapp/03261907.log for details

                       

                       

                      Exiting with status 0

                       

                       

                      Stopped Database....

                       

                       

                      Now started Database

                       

                      started OK

                       

                      Now started Application

                      applprod@cloapp:/home/applprod>./startup.sh

                       

                       

                      You are running adstrtal.sh version 115.14

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adapcctl.sh start

                      script returned:

                      ****************************************************

                       

                       

                      adapcctl.sh version 115.43

                       

                       

                      Starting Apache Web Server Listener (dedicated HTTP) ...

                      Starting Apache Web Server Listener (dedicated PLSQL) ...

                       

                       

                      adapcctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adalnctl.sh start

                      script returned:

                      ****************************************************

                       

                       

                      adalnctl.sh version

                       

                       

                      Checking for FNDFS executable.

                      Starting listener process APPS_PROD.

                       

                       

                      adalnctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adcmctl.sh start

                      script returned:

                      ****************************************************

                       

                       

                      You are running adcmctl.sh version 115.19

                       

                       

                      Starting concurrent manager for PROD ...

                      Starting PROD_0326@PROD Internal Concurrent Manager

                      Default printer is noprint

                       

                       

                      adcmctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adfrmctl.sh start

                      script returned:

                      ****************************************************

                       

                       

                      You are running adfrmctl.sh version 115.33

                       

                       

                      Starting forms server for PROD on port 9000.

                       

                       

                      adfrmctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adrepctl.sh start

                      script returned:

                      ****************************************************

                       

                       

                      You are running adrepctl.sh version 115.29

                       

                       

                      starting Reports Server for PROD on port 7000.

                       

                       

                      adrepctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adfmcctl.sh start

                      script returned:

                      ****************************************************

                       

                       

                      You are running adfmcctl.sh version 115.16

                       

                       

                      Starting forms load balancing client for PROD.

                       

                       

                      adfmcctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/adfmsctl.sh start

                      script returned:

                      ****************************************************

                       

                       

                      You are running adfmsctl.sh version 115.12

                       

                       

                      starting forms  metrics server for PROD.

                       

                       

                      adfmsctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/jtffmctl.sh start

                      script returned:

                      ****************************************************

                       

                       

                      You are running jtffmctl.sh version 115.7

                       

                       

                      Starting Fulfillment Server for PROD on port 9300 ...

                       

                       

                      jtffmctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      Executing service control script:

                      /oraapp/prodcomn/admin/scripts/PROD_cloapp/addisctl.sh start

                      script returned:

                      ****************************************************

                       

                       

                      addisctl.sh version 115.8

                       

                       

                      /oraapp/prodora/8.0.6/vbroker/bin/osagent

                      Started osagent.

                      Osagent logs messages to the file /oraapp/prodora/8.0.6/discwb4/util/osagent.log.

                      Waiting for OAD to start...

                      Started OAD.

                      OAD logs messages to the file /oraapp/prodora/8.0.6/discwb4/util/oad.log.

                      Discoverer Locator Started.

                      Locator logs messages to the file /oraapp/prodora/8.0.6/discwb4/util/locator.log.

                       

                       

                      addisctl.sh: exiting with status 0

                       

                       

                       

                       

                      .end std out.

                       

                       

                      .end err out.

                       

                       

                      ****************************************************

                       

                       

                       

                       

                      All enabled services for this node are started.

                      Check logfile /oraapp/prodcomn/admin/log/PROD_cloapp/03261910.log for details

                       

                       

                      Exiting with status 0

                       

                      Checking the log file for concurrent manager

                       

                      applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>cat PROD_0326.mgr

                       

                       

                      ========================================================================

                      Starting PROD_0326@PROD Internal Concurrent Manager -- shell process ID 422094

                       

                       

                                logfile=/oraapp/prodcomn/admin/log/PROD_cloapp/PROD_0326.mgr

                                PRINTER=noprint

                                 mailto=applprod

                                restart=N

                                   diag=N

                                  sleep=60 (default)

                                   pmon=20 (default)

                                 quesiz=1  (default)

                       

                       

                      establish_icm - Unable to insert ICM record

                      Call to establish_icm failed

                      The Internal Concurrent Manager has encountered an error.

                       

                       

                      Review concurrent manager log file for more detailed information. : 26-MAR-2014 15:10:12 -

                       

                       

                       

                       

                      Shutting down Internal Concurrent Manager : 26-MAR-2014 15:10:12

                       

                       

                      List of errors encountered:

                      .............................................................................

                       

                       

                      _ 1 _

                      Routine AFPCSQ encountered an ORACLE error.  .

                       

                       

                      Review your error messages for the cause of the error. (=<POINTER>)

                      .............................................................................

                       

                       

                       

                       

                      APP-FND-01564: ORACLE error 1653 in insert_icm_record

                       

                       

                      Cause:  insert_icm_record failed due to ORA-01653: unable to extend table APPLSYS.FND_CONCURRENT_PROCESSES by 65 in tablespace APPLSYSD.

                       

                       

                      The SQL statement being executed at the time of the error was: INSERT   INTO FND_CONCURRENT_PROCESSES       (CONCURRENT_PROCESS_ID, LAST_UPDATE_DATE,        CREATION_DATE,         CREATED_BY,        LAST_UPDATED_BY,       ORACLE_PROCESS_ID,        QUEUE_APPLICATION_ID,  CONCURRENT_QUEUE_ID,        OS_PROCESS_ID,         SESSION_ID,        PROCESS_START_DATE,    PROCESS_STATUS_CODE,        MANAGER_TYPE,          NODE_NAME,        DB_NAME,               DB_DOMAIN,        DB_INSTANCE, Instance_Number) VALUES       (:cpid,                 SYSDATE,        SYSDATE,               :mgrusrid,        :mgrusrid,             :opid,        :qappid,               :qid,        :ospid,                :osid,        SYSDATE,               'A',        0,                     :node,        :dbname,               :dbdomain,        :dbinst, (Select instance_number from v$instance)) and was executed from the file &ERRFILE.

                       

                       

                      List of errors encountered:

                      .............................................................................

                       

                       

                      _ 1 _

                      Routine AFPCAL received failure code while parsing or running your

                      concurrent program CPMGR

                       

                       

                      Review your concurrent request log file for more detailed information.

                      Make sure you are passing arguments in the correct format.

                      .............................................................................

                       

                       

                      The PROD_0326@PROD internal concurrent manager has terminated with status 1 - giving up.

                       

                       

                      ========================================================================

                      Starting PROD_0326@PROD Internal Concurrent Manager -- shell process ID 340138

                       

                       

                                logfile=/oraapp/prodcomn/admin/log/PROD_cloapp/PROD_0326.mgr

                                PRINTER=noprint

                                 mailto=applprod

                                restart=N

                                   diag=N

                                  sleep=60 (default)

                                   pmon=20 (default)

                                 quesiz=1  (default)

                       

                       

                      The Internal Concurrent Manager has encountered an error.

                       

                       

                      Review concurrent manager log file for more detailed information. : 26-MAR-2014 19:11:37 -

                       

                       

                       

                       

                      Shutting down Internal Concurrent Manager : 26-MAR-2014 19:11:37

                       

                       

                      List of errors encountered:

                      .............................................................................

                       

                       

                      _ 1 _

                      Concurrent Manager cannot find error description for CONC-System Node

                      Name not Registered

                       

                       

                      Contact your support representative.

                      .............................................................................

                       

                       

                       

                       

                       

                       

                      List of errors encountered:

                      .............................................................................

                       

                       

                      _ 1 _

                      Routine AFPCAL received failure code while parsing or running your

                      concurrent program CPMGR

                       

                       

                      Review your concurrent request log file for more detailed information.

                      Make sure you are passing arguments in the correct format.

                      .............................................................................

                       

                       

                      The PROD_0326@PROD internal concurrent manager has terminated with status 1 - giving up.

                      applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>

                       

                       

                      still the same error even after adding space and extending the tablespace....

                      • 8. Re: Concurrent Manager starts but doesn't work
                        Hussein Sawwan-Oracle
                        Cause:  insert_icm_record failed due to ORA-01653: unable to extend table APPLSYS.FND_CONCURRENT_PROCESSES by 65 in tablespace APPLSYSD.

                         

                        You still need to add more space to this tablespace (APPLSYSD).

                         

                        For adrepctl.sql, once you fix the space issue run AutoConfig on the apps tier node and make sure it completes successfully.

                         

                        Thanks,

                        Hussein

                        1 person found this helpful
                        • 9. Re: Concurrent Manager starts but doesn't work
                          Shakti1807

                          hi,

                          You may following note ids helpful.

                           

                          ICM Not Starting: Node Name not Registered (Doc ID 602942.1)

                          All Requests Have a Status of Pending/No Manager After Power Outage (Doc ID 160209.1)

                          Basically, you first need to check host name and then only stop managers, verify all manager processes are gone then run cmclean.sql and try to start managers.

                           

                          Thanks,

                          Shakti

                          1 person found this helpful
                          • 10. Re: Concurrent Manager starts but doesn't work
                            Hussein Sawwan-Oracle

                            Shakti,


                            The tablespace size issue should be resolved first before proceeding with any other action plan.


                            Thanks,

                            Hussein

                            1 person found this helpful
                            • 11. Re: Concurrent Manager starts but doesn't work
                              Shakti1807

                              Yeah I agree that and I assumed that @984556 has checked the space available in APPLSYSD after adding a datafile of 1G.

                              Also, note id 160209.1 suggest to run cmclean for the somewhat similar issue.. (index is referred there)

                               

                              @984556 can you please check the space available in APPLSYSD tablespace and paste here..

                               

                              Thanks,

                              Shakti

                              1 person found this helpful
                              • 12. Re: Concurrent Manager starts but doesn't work
                                984556

                                As requested by Hussein Sawwan & Shakti

                                 

                                1) Resolve the tablespace issue

                                 

                                * Resolved , added enough space in the tablespace APPLSYSD

                                 

                                Next, started the Apps,

                                 

                                now there is a different error with the Concurrent Manager

                                 

                                applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>cat PROD_0327.mgr

                                 

                                 

                                ========================================================================

                                Starting PROD_0327@PROD Internal Concurrent Manager -- shell process ID 450562

                                 

                                 

                                          logfile=/oraapp/prodcomn/admin/log/PROD_cloapp/PROD_0327.mgr

                                          PRINTER=noprint

                                           mailto=applprod

                                          restart=N

                                             diag=N

                                            sleep=60 (default)

                                             pmon=20 (default)

                                           quesiz=1  (default)

                                 

                                 

                                The Internal Concurrent Manager has encountered an error.

                                 

                                 

                                Review concurrent manager log file for more detailed information. : 27-MAR-2014 13:38:15 -

                                 

                                 

                                 

                                 

                                Shutting down Internal Concurrent Manager : 27-MAR-2014 13:38:15

                                 

                                 

                                List of errors encountered:

                                .............................................................................

                                 

                                 

                                _ 1 _

                                Concurrent Manager cannot find error description for CONC-System Node

                                Name not Registered

                                 

                                 

                                Contact your support representative.

                                .............................................................................

                                 

                                 

                                 

                                 

                                 

                                 

                                List of errors encountered:

                                .............................................................................

                                 

                                 

                                _ 1 _

                                Routine AFPCAL received failure code while parsing or running your

                                concurrent program CPMGR

                                 

                                 

                                Review your concurrent request log file for more detailed information.

                                Make sure you are passing arguments in the correct format.

                                .............................................................................

                                 

                                 

                                The PROD_0327@PROD internal concurrent manager has terminated with status 1 - giving up.

                                applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>

                                 

                                 

                                as Hussein mentioned in another post

                                 

                                Please see these docs.

                                 

                                Concurrent Manager Cannot Find Error Description for CONC-System Node : Name not Registered [ID 375813.1]

                                Unable To Start Concurrent Manager And Failing With Message Concurrent Manager cannot find error description for CONC-Get plsql file names [ID 1161386.1]

                                Concurrent Processing - CONC-SM TNS FAIL In Internal Concurrent Manager (ICM) Log File With Virtual Hosts Configuration [ID 961216.1]

                                Conc-System Node Name Not Registered After Fresh Install Using Virtual Name [ID 948644.1]

                                ICM Not Starting: Node Name not Registered [ID 602942.1]

                                Why are the Concurrent Managers Failing with 'CONC-System Node Name not Registered" Error After Applying Patch 6461517? [ID 1108452.1]


                                ----


                                But i had run ccm.sql and cmclean.sql before, should I re run it again ?



                                This is what I did


                                oraprod@clodb:/home/oraprod>cd /oratech/proddb/9.2.0/

                                oraprod@clodb:/oratech/proddb/9.2.0>. ./PROD_clodb.env

                                oraprod@clodb:/oratech/proddb/9.2.0>sqlplus apps/apps

                                 

                                 

                                SQL*Plus: Release 9.2.0.3.0 - Production on Thu Mar 27 16:54:57 2014

                                 

                                 

                                Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

                                 

                                 

                                 

                                 

                                Connected to:

                                Oracle9i Enterprise Edition Release 9.2.0.3.0 - 64bit Production

                                With the Partitioning, OLAP and Oracle Data Mining options

                                JServer Release 9.2.0.3.0 - Production

                                 

                                 

                                SQL>

                                 

                                SQL> select node_name from fnd_nodes;

                                 

                                 

                                no rows selected

                                 

                                 

                                SQL>

                                 



                                searching on the forum, someone mentioned


                                1. Verify correct node name is registered.

                                - Login with sysadmin responsibility

                                 

                                - Navigate to Install > Nodes


                                { i see there is nothing ... }

                                apps server

                                root@cloapp:/>uname -a

                                AIX cloapp 3 5 00C7780E4C00

                                DB  server

                                oraprod@clodb:/oratech/proddb/9.2.0>uname -a

                                AIX clodb 1 6 00C7780E4C00

                                 

                                2. Enter correct name and save the change.


                                I entered

                                cloapp     IBM AIX   IP Address 10.1.1.120

                                save and proceed


                                One record saved

                                Node ID is 63


                                 

                                3. Restart concurrent managers

                                {restarted}




                                Now on the database


                                SQL> select node_name from fnd_nodes;

                                 

                                 

                                NODE_NAME

                                ------------------------------

                                CLOAPP

                                 

                                 

                                SQL>

                                 

                                 

                                 

                                Did I enter the node name correctly ?

                                 

                                 

                                After restarting, I checked concurrent manager log

                                 

                                this was found

                                 

                                applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>cat PROD_0327.mgr

                                 

                                 

                                ========================================================================

                                Starting PROD_0327@PROD Internal Concurrent Manager -- shell process ID 450562

                                 

                                 

                                          logfile=/oraapp/prodcomn/admin/log/PROD_cloapp/PROD_0327.mgr

                                          PRINTER=noprint

                                           mailto=applprod

                                          restart=N

                                             diag=N

                                            sleep=60 (default)

                                             pmon=20 (default)

                                           quesiz=1  (default)

                                 

                                 

                                The Internal Concurrent Manager has encountered an error.

                                 

                                 

                                Review concurrent manager log file for more detailed information. : 27-MAR-2014 13:38:15 -

                                 

                                 

                                 

                                 

                                Shutting down Internal Concurrent Manager : 27-MAR-2014 13:38:15

                                 

                                 

                                List of errors encountered:

                                .............................................................................

                                 

                                 

                                _ 1 _

                                Concurrent Manager cannot find error description for CONC-System Node

                                Name not Registered

                                 

                                 

                                Contact your support representative.

                                .............................................................................

                                 

                                 

                                 

                                 

                                 

                                 

                                List of errors encountered:

                                .............................................................................

                                 

                                 

                                _ 1 _

                                Routine AFPCAL received failure code while parsing or running your

                                concurrent program CPMGR

                                 

                                 

                                Review your concurrent request log file for more detailed information.

                                Make sure you are passing arguments in the correct format.

                                .............................................................................

                                 

                                 

                                The PROD_0327@PROD internal concurrent manager has terminated with status 1 - giving up.

                                applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>cat PROD_0327.mgr

                                 

                                 

                                ========================================================================

                                Starting PROD_0327@PROD Internal Concurrent Manager -- shell process ID 450562

                                 

                                 

                                          logfile=/oraapp/prodcomn/admin/log/PROD_cloapp/PROD_0327.mgr

                                          PRINTER=noprint

                                           mailto=applprod

                                          restart=N

                                             diag=N

                                            sleep=60 (default)

                                             pmon=20 (default)

                                           quesiz=1  (default)

                                 

                                 

                                The Internal Concurrent Manager has encountered an error.

                                 

                                 

                                Review concurrent manager log file for more detailed information. : 27-MAR-2014 13:38:15 -

                                 

                                 

                                 

                                 

                                Shutting down Internal Concurrent Manager : 27-MAR-2014 13:38:15

                                 

                                 

                                List of errors encountered:

                                .............................................................................

                                 

                                 

                                _ 1 _

                                Concurrent Manager cannot find error description for CONC-System Node

                                Name not Registered

                                 

                                 

                                Contact your support representative.

                                .............................................................................

                                 

                                 

                                 

                                 

                                 

                                 

                                List of errors encountered:

                                .............................................................................

                                 

                                 

                                _ 1 _

                                Routine AFPCAL received failure code while parsing or running your

                                concurrent program CPMGR

                                 

                                 

                                Review your concurrent request log file for more detailed information.

                                Make sure you are passing arguments in the correct format.

                                .............................................................................

                                 

                                 

                                The PROD_0327@PROD internal concurrent manager has terminated with status 1 - giving up.

                                 

                                 

                                ========================================================================

                                Starting PROD_0327@PROD Internal Concurrent Manager -- shell process ID 479282

                                 

                                 

                                          logfile=/oraapp/prodcomn/admin/log/PROD_cloapp/PROD_0327.mgr

                                          PRINTER=noprint

                                           mailto=applprod

                                          restart=N

                                             diag=N

                                            sleep=60 (default)

                                             pmon=20 (default)

                                           quesiz=1  (default)

                                 

                                 

                                 

                                 

                                +---------------------------------------------------------------------------+

                                Application Object Library: Concurrent Processing version 11.5

                                 

                                 

                                Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.

                                 

                                 

                                Internal Concurrent Manager started : 27-MAR-2014 14:28:37

                                 

                                 

                                +---------------------------------------------------------------------------+

                                 

                                 

                                                   Process monitor session started : 27-MAR-2014 14:28:38

                                 

                                 

                                Starting FNDCRM Concurrent Manager                 : 27-MAR-2014 14:28:38

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager FNDCRM with library /oraapp/prodappl/fnd/11.5.0/bin/FNDCRM.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admin : 27-MAR-2014 14:28:38

                                 

                                 

                                Starting FNDSCH Concurrent Manager                 : 27-MAR-2014 14:28:38

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager FNDSCH with library /oraapp/prodappl/fnd/11.5.0/bin/FNDSCH.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admin : 27-MAR-2014 14:28:38

                                 

                                 

                                Starting STANDARD Concurrent Manager               : 27-MAR-2014 14:28:38

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager STANDARD with library /oraapp/prodappl/fnd/11.5.0/bin/FNDLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system ad : 27-MAR-2014 14:28:38

                                Starting STANDARD Concurrent Manager               : 27-MAR-2014 14:28:38

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager STANDARD with library /oraapp/prodappl/fnd/11.5.0/bin/FNDLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system ad : 27-MAR-2014 14:28:38

                                Starting STANDARD Concurrent Manager               : 27-MAR-2014 14:28:38

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager STANDARD with library /oraapp/prodappl/fnd/11.5.0/bin/FNDLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system ad : 27-MAR-2014 14:28:38

                                Starting STANDARD Concurrent Manager               : 27-MAR-2014 14:28:38

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager STANDARD with library /oraapp/prodappl/fnd/11.5.0/bin/FNDLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system ad : 27-MAR-2014 14:28:38

                                Starting STANDARD Concurrent Manager               : 27-MAR-2014 14:28:38

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager STANDARD with library /oraapp/prodappl/fnd/11.5.0/bin/FNDLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system ad : 27-MAR-2014 14:28:38

                                Starting STANDARD Concurrent Manager               : 27-MAR-2014 14:28:38

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager STANDARD with library /oraapp/prodappl/fnd/11.5.0/bin/FNDLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system ad : 27-MAR-2014 14:28:38

                                 

                                 

                                Starting PODAMGR Concurrent Manager                : 27-MAR-2014 14:28:38

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager PODAMGR with library /oraapp/prodappl/po/11.5.0/bin/POXCON.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admin : 27-MAR-2014 14:28:38

                                Starting PODAMGR Concurrent Manager                : 27-MAR-2014 14:28:38

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager PODAMGR with library /oraapp/prodappl/po/11.5.0/bin/POXCON.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admin : 27-MAR-2014 14:28:38

                                Starting PODAMGR Concurrent Manager                : 27-MAR-2014 14:28:38

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager PODAMGR with library /oraapp/prodappl/po/11.5.0/bin/POXCON.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admin : 27-MAR-2014 14:28:38

                                 

                                 

                                Starting RCVOLTM Concurrent Manager                : 27-MAR-2014 14:28:38

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager RCVOLTM with library /oraapp/prodappl/po/11.5.0/bin/RCVOLTM.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admi : 27-MAR-2014 14:28:38

                                Starting RCVOLTM Concurrent Manager                : 27-MAR-2014 14:28:38

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager RCVOLTM with library /oraapp/prodappl/po/11.5.0/bin/RCVOLTM.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admi : 27-MAR-2014 14:28:38

                                Starting RCVOLTM Concurrent Manager                : 27-MAR-2014 14:28:38

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager RCVOLTM with library /oraapp/prodappl/po/11.5.0/bin/RCVOLTM.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admi : 27-MAR-2014 14:28:38

                                 

                                 

                                Starting PASMGR Concurrent Manager                 : 27-MAR-2014 14:28:38

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager PASMGR with library /oraapp/prodappl/pa/11.5.0/bin/PALIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admini : 27-MAR-2014 14:28:38

                                 

                                 

                                Starting OAMCOLMGR Concurrent Manager              : 27-MAR-2014 14:28:38

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager OAMCOLMGR with library /oraapp/prodappl/fnd/11.5.0/bin/FNDLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system a : 27-MAR-2014 14:28:39

                                 

                                 

                                Starting INVMGR Concurrent Manager                 : 27-MAR-2014 14:28:39

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager INVMGR with library /oraapp/prodappl/inv/11.5.0/bin/INVLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admi : 27-MAR-2014 14:28:39

                                Starting INVMGR Concurrent Manager                 : 27-MAR-2014 14:28:39

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager INVMGR with library /oraapp/prodappl/inv/11.5.0/bin/INVLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admi : 27-MAR-2014 14:28:39

                                Starting INVMGR Concurrent Manager                 : 27-MAR-2014 14:28:39

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager INVMGR with library /oraapp/prodappl/inv/11.5.0/bin/INVLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system admi : 27-MAR-2014 14:28:39

                                 

                                 

                                Starting IEU_SH_CS Concurrent Manager              : 27-MAR-2014 14:28:39

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager IEU_SH_CS with library /oraapp/prodappl/fnd/11.5.0/bin/FNDLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system a : 27-MAR-2014 14:28:39

                                 

                                 

                                Starting IEU_WL_CS Concurrent Manager              : 27-MAR-2014 14:28:39

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Routine AFPEIM encountered an error while starting concurrent manager IEU_WL_CS with library /oraapp/prodappl/fnd/11.5.0/bin/FNDLIBR.

                                 

                                 

                                Check that your system has enough resources to start a concurrent manager process.  Contact your system a : 27-MAR-2014 14:28:39

                                 

                                 

                                Starting WFALSNRSVC Concurrent Manager             : 27-MAR-2014 14:28:39

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Found dead process: spid=(999999), cpid=(85712), Service Instance=(1139)

                                 

                                 

                                Starting WFMLRSVC Concurrent Manager               : 27-MAR-2014 14:28:39

                                 

                                 

                                Could not start Service Manager FNDSM_CLOAPP_PROD. The TNS alias could not be located, the listener process on CLOAPP could not be contacted, or the listener failed to spawn the Service Manager process.

                                Found dead process: spid=(999999), cpid=(85713), Service Instance=(1140)

                                                     Process monitor session ended : 27-MAR-2014 14:28:39

                                applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>

                                • 13. Re: Concurrent Manager starts but doesn't work
                                  Shakti1807

                                  I am not sure why are you missing fndsm tns entry. Autoconfig should have created the tnsnames.ora having this entry.

                                   

                                  Assuming the architecture you have is

                                  clodb - only db tier

                                  cloapp - all application (cm,forms,apache)

                                   

                                  1.Check if you have the correct tns entry for FNDSM_CLOAPP_PROD in tnsnames.ora on app tier

                                  2. Make sure apps listener is running

                                  If it does not work, the cleanest way for it would be to:

                                   

                                  1. Stop all apps services

                                  2. Run exec fnd_conc_clone.setup_clean(); as apps

                                  3. Run autoconfig on dbtier

                                  4. Run autoconfig on apps tier

                                  Check fnd_nodes, you should have entries for both the nodes, and check tnsnames.ora, should have correct tns entry.

                                  5. Start services.

                                   

                                  Thanks,

                                  Shakti

                                  • 14. Re: Concurrent Manager starts but doesn't work
                                    984556

                                    Shakti,

                                     

                                    As you asked:

                                     

                                    1. Stop all apps services

                                     

                                    Stopped Apps.

                                     

                                    2. Run exec fnd_conc_clone.setup_clean(); as apps

                                     

                                    oraprod@clodb:/oratech/proddb/9.2.0>sqlplus apps/apps

                                     

                                     

                                    SQL*Plus: Release 9.2.0.3.0 - Production on Thu Mar 27 22:54:28 2014

                                     

                                     

                                    Copyright (c) 1982, 2002, Oracle Corporation.  All rights reserved.

                                     

                                     

                                     

                                     

                                    Connected to:

                                    Oracle9i Enterprise Edition Release 9.2.0.3.0 - 64bit Production

                                    With the Partitioning, OLAP and Oracle Data Mining options

                                    JServer Release 9.2.0.3.0 - Production

                                     

                                     

                                    SQL> EXEC FND_CONC_CLONE.SETUP_CLEAN;

                                     

                                     

                                    PL/SQL procedure successfully completed.

                                     

                                     

                                    SQL> commit;

                                     

                                     

                                    Commit complete.

                                     

                                     

                                    SQL> exit

                                     

                                    3. Run autoconfig on dbtier

                                     

                                    oraprod@clodb:/oratech/proddb/9.2.0>2.0/appsutil/scripts/PROD_clodb          <

                                    oraprod@clodb:/oratech/proddb/9.2.0/appsutil/scripts/PROD_clodb>ls *.sh

                                    adautocfg.sh  addbctl.sh    addlnctl.sh   adlsnodes.sh

                                    oraprod@clodb:/oratech/proddb/9.2.0/appsutil/scripts/PROD_clodb>ocfg.sh      <

                                    Enter the APPS user password:

                                    AutoConfig is configuring the Database environment...

                                     

                                     

                                    AutoConfig will consider the custom templates if present.

                                            Using ORACLE_HOME location : /oratech/proddb/9.2.0

                                            Using java command          : /oratech/proddb/9.2.0/jdk/bin/java

                                            Java version                : 1.3.1

                                            Classpath                   : /oratech/proddb/9.2.0/jdk/jre/lib/rt.jar:/oratech/proddb/9.2.0/jdk/jre/lib/i18n.jar:/oratech/proddb/9.2.0/jdk/lib/dt.jar:/oratech/proddb/9.2.0/jdk/lib/tools.jar:/oratech/proddb/9.2.0/jdbc/lib/classes12.jar:/oratech/proddb/9.2.0/appsutil/java/xmlparserv2.zip:/oratech/proddb/9.2.0/appsutil/java:/oratech/proddb/9.2.0/jlib/netcfg.jar

                                     

                                     

                                            Using Context file          : /oratech/proddb/9.2.0/appsutil/PROD_clodb.xml

                                     

                                     

                                    Context Value Management will now update the Context file

                                     

                                     

                                            Updating Context file...COMPLETED

                                     

                                     

                                            Attempting upload of Context file and templates to database...COMPLETED

                                     

                                     

                                    Updating rdbms version in Context file to db920

                                    Updating rdbms type in Context file to 64 bits

                                    Configuring templates from ORACLE_HOME ...

                                     

                                     

                                    AutoConfig completed successfully.

                                    The log file for this session is located at: /oratech/proddb/9.2.0/appsutil/log/PROD_clodb/03272257/adconfig.log

                                    oraprod@clodb:/oratech/proddb/9.2.0/appsutil/scripts/PROD_clodb>

                                     

                                    4. Run autoconfig on apps tier

                                     

                                    applprod@cloapp:/oraapp/prodcomn/admin/scripts/PROD_cloapp>./adautocfg.sh

                                    Enter the APPS user password:

                                    AutoConfig is configuring the Applications environment...

                                     

                                     

                                    AutoConfig will consider the custom templates if present.

                                            Using APPL_TOP location     : /oraapp/prodappl

                                            Using java command          : /usr/java131/bin/java

                                            Java version                : 1.3.1

                                            Classpath                   : /usr/java131/jre/lib/rt.jar:/usr/java131/jre/lib/i18n.jar:/usr/java131/lib/dt.jar:/usr/java131/lib/tools.jar:/oraapp/prodcomn/java/appsborg2.zip:/oraapp/prodcomn/java

                                     

                                     

                                            Using Context file          : /oraapp/prodappl/admin/PROD_cloapp.xml

                                     

                                     

                                    Context Value Management will now update the Context file

                                     

                                     

                                            Updating Context file...COMPLETED

                                     

                                     

                                            Attempting upload of Context file and templates to database...COMPLETED

                                     

                                     

                                    Configuring templates from all of the product tops...

                                            Configuring AD_TOP........COMPLETED

                                            Configuring FND_TOP.......COMPLETED

                                            Configuring ICX_TOP.......COMPLETED

                                            Configuring IEO_TOP.......COMPLETED

                                            Configuring ABM_TOP.......COMPLETED

                                            Configuring ECX_TOP.......COMPLETED

                                            Configuring BIS_TOP.......COMPLETED

                                            Configuring GL_TOP........COMPLETED

                                            Configuring AMS_TOP.......COMPLETED

                                            Configuring WSH_TOP.......COMPLETED

                                            Configuring OKE_TOP.......COMPLETED

                                            Configuring OKL_TOP.......COMPLETED

                                            Configuring OKS_TOP.......COMPLETED

                                            Configuring CSF_TOP.......COMPLETED

                                            Configuring XNC_TOP.......COMPLETED

                                            Configuring IGS_TOP.......COMPLETED

                                            Configuring IBY_TOP.......COMPLETED

                                            Configuring PA_TOP........COMPLETED

                                            Configuring JTF_TOP.......COMPLETED

                                            Configuring MWA_TOP.......COMPLETED

                                            Configuring CN_TOP........COMPLETED

                                            Configuring CSI_TOP.......COMPLETED

                                            Configuring WIP_TOP.......COMPLETED

                                            Configuring CSE_TOP.......COMPLETED

                                            Configuring EAM_TOP.......COMPLETED

                                            Configuring IMT_TOP.......COMPLETED

                                            Configuring FTE_TOP.......COMPLETED

                                            Configuring ONT_TOP.......COMPLETED

                                            Configuring AR_TOP........COMPLETED

                                            Configuring AHL_TOP.......COMPLETED

                                            Configuring CSD_TOP.......COMPLETED

                                            Configuring IGC_TOP.......COMPLETED

                                     

                                     

                                    AutoConfig completed successfully.

                                    The log file for this session is located at: /oraapp/prodappl/admin/PROD_cloapp/log/03271959/adconfig.log

                                    applprod@cloapp:/oraapp/prodcomn/admin/scripts/PROD_cloapp>

                                     

                                    * Check fnd_nodes, you should have entries for both the nodes, and check tnsnames.ora, should have correct tns entry.

                                     

                                    5. Start services.

                                     

                                    started apps

                                     

                                    checking the concurrent manager log

                                     

                                    ========================================================================

                                    Starting PROD_0327@PROD Internal Concurrent Manager -- shell process ID 6750430

                                     

                                     

                                              logfile=/oraapp/prodcomn/admin/log/PROD_cloapp/PROD_0327.mgr

                                              PRINTER=noprint

                                               mailto=applprod

                                              restart=N

                                                 diag=N

                                                sleep=60 (default)

                                                 pmon=20 (default)

                                               quesiz=1  (default)

                                     

                                     

                                    Found dead process: spid=(491536), cpid=(87371), ORA pid=(51), manager=(0/4)

                                    Found dead process: spid=(417956), cpid=(87373), ORA pid=(52), manager=(0/5)

                                    Found dead process: spid=(483512), cpid=(87374), ORA pid=(42), manager=(0/0)

                                    Found dead process: spid=(438280), cpid=(87375), ORA pid=(43), manager=(0/0)

                                    Found dead process: spid=(409812), cpid=(87376), ORA pid=(44), manager=(0/0)

                                    Found dead process: spid=(331868), cpid=(87377), ORA pid=(45), manager=(0/0)

                                    Found dead process: spid=(352364), cpid=(87378), ORA pid=(75), manager=(0/0)

                                    Found dead process: spid=(499808), cpid=(87379), ORA pid=(46), manager=(0/0)

                                    Found dead process: spid=(450566), cpid=(87380), ORA pid=(55), manager=(201/15)

                                    Found dead process: spid=(442526), cpid=(87381), ORA pid=(54), manager=(201/15)

                                    Found dead process: spid=(217334), cpid=(87382), ORA pid=(56), manager=(201/15)

                                    Found dead process: spid=(364576), cpid=(87383), ORA pid=(57), manager=(201/222)

                                    Found dead process: spid=(508152), cpid=(87384), ORA pid=(58), manager=(201/222)

                                    Found dead process: spid=(512250), cpid=(87385), ORA pid=(59), manager=(201/222)

                                    Found dead process: spid=(516348), cpid=(87386), ORA pid=(65), manager=(275/40)

                                    Found dead process: spid=(520446), cpid=(87387), ORA pid=(48), manager=(0/1044)

                                    Found dead process: spid=(524288), cpid=(87388), ORA pid=(61), manager=(401/10)

                                    Found dead process: spid=(528386), cpid=(87389), ORA pid=(63), manager=(401/10)

                                    Found dead process: spid=(532484), cpid=(87390), ORA pid=(62), manager=(401/10)

                                    Found dead process: spid=(536582), cpid=(87391), ORA pid=(49), manager=(696/1024)

                                    Found dead process: spid=(540680), cpid=(87392), ORA pid=(50), manager=(696/1025)

                                    Found dead process: spid=(434354), cpid=(87370), ORA pid=(13), manager=(0/1)

                                     

                                     

                                    +---------------------------------------------------------------------------+

                                    Application Object Library: Concurrent Processing version 11.5

                                     

                                     

                                    Copyright (c) 1979, 1999, Oracle Corporation. All rights reserved.

                                     

                                     

                                    Internal Concurrent Manager started : 27-MAR-2014 20:07:57

                                     

                                     

                                    +---------------------------------------------------------------------------+

                                     

                                     

                                                       Process monitor session started : 27-MAR-2014 20:08:03

                                    Found dead process: spid=(557112), cpid=(87453), Service Instance=(1139)

                                    Found dead process: spid=(569546), cpid=(87454), Service Instance=(1140)

                                     

                                     

                                    Starting FNDCRM Concurrent Manager                 : 27-MAR-2014 20:08:08

                                     

                                     

                                     

                                     

                                    Starting FNDSCH Concurrent Manager                 : 27-MAR-2014 20:08:08

                                     

                                     

                                     

                                     

                                    Starting STANDARD Concurrent Manager               : 27-MAR-2014 20:08:08

                                    Starting STANDARD Concurrent Manager               : 27-MAR-2014 20:08:08

                                    Starting STANDARD Concurrent Manager               : 27-MAR-2014 20:08:08

                                    Starting STANDARD Concurrent Manager               : 27-MAR-2014 20:08:08

                                    Starting STANDARD Concurrent Manager               : 27-MAR-2014 20:08:08

                                    Starting STANDARD Concurrent Manager               : 27-MAR-2014 20:08:08

                                     

                                     

                                     

                                     

                                    Starting PODAMGR Concurrent Manager                : 27-MAR-2014 20:08:08

                                    Starting PODAMGR Concurrent Manager                : 27-MAR-2014 20:08:08

                                    Starting PODAMGR Concurrent Manager                : 27-MAR-2014 20:08:09

                                     

                                     

                                     

                                     

                                    Starting RCVOLTM Concurrent Manager                : 27-MAR-2014 20:08:09

                                    Starting RCVOLTM Concurrent Manager                : 27-MAR-2014 20:08:09

                                    Starting RCVOLTM Concurrent Manager                : 27-MAR-2014 20:08:09

                                     

                                     

                                     

                                     

                                    Starting PASMGR Concurrent Manager                 : 27-MAR-2014 20:08:09

                                     

                                     

                                     

                                     

                                    Starting OAMCOLMGR Concurrent Manager              : 27-MAR-2014 20:08:09

                                     

                                     

                                     

                                     

                                    Starting INVMGR Concurrent Manager                 : 27-MAR-2014 20:08:09

                                    Starting INVMGR Concurrent Manager                 : 27-MAR-2014 20:08:09

                                    Starting INVMGR Concurrent Manager                 : 27-MAR-2014 20:08:09

                                     

                                     

                                     

                                     

                                    Starting IEU_SH_CS Concurrent Manager              : 27-MAR-2014 20:08:09

                                     

                                     

                                     

                                     

                                    Starting IEU_WL_CS Concurrent Manager              : 27-MAR-2014 20:08:09

                                     

                                     

                                     

                                     

                                    Starting WFALSNRSVC Concurrent Manager             : 27-MAR-2014 20:08:09

                                     

                                     

                                     

                                     

                                    Starting WFMLRSVC Concurrent Manager               : 27-MAR-2014 20:08:09

                                     

                                     

                                                         Process monitor session ended : 27-MAR-2014 20:08:10

                                    applprod@cloapp:/oraapp/prodcomn/admin/log/PROD_cloapp>

                                     

                                     

                                     

                                     

                                    Thank you. Problem resolved.