8 Replies Latest reply on Feb 20, 2016 4:11 PM by Karthik Appsys

    R12.1.3 ICM going down - Actual Target mismatch

    Karthik Appsys

      Dear Legends,

       

      We recently did a upgrade of R12.0.6 to R12.1.3 with DB as from 10.2.0.3 to 12.1.0.2 and applied RPC4 on top of it. From last 2 weeks once in 7 days the ICM is going down due to the following error.

       

      The Internal Concurrent Manager has encountered an error.

       

      Review concurrent manager log file for more detailed information. : 17-FEB-2016 13:20:00 -

       

       

      Shutting down Internal Concurrent Manager : 17-FEB-2016 13:20:00

       

      List of errors encountered:

      .............................................................................

      _ 1 _

      Routine FDPCRQ encountered an ORACLE error. ORA-04031: unable to

      allocate 40 bytes of shared memory ("shared pool".

       

      Review your error messages for the cause of the error. (=<POINTER>)

      .............................................................................

      List of errors encountered:

      .............................................................................

      _ 1 _

      Routine AFPCAL received failure code while parsing or running your

      concurrent program CPMGR

       

      Review your concurrent request log file for more detailed information.

      Make sure you are passing arguments in the correct format.

      .............................................................................

       

      The PROD_0217@PROD internal concurrent manager has terminated with status 1 - giving up.

       

       

      And then we checked the Actuals and Targets value are having different like 0 1, so we did a adcmctl.sh stop, but it took time to stop so we issued

      killall FNDLIBR then all process of FNDLIBR has been killed.

      After this we issued ./adcmctl.sh start all concurrents have been started and Actuals and Targets are holding same value like 1 1 with the following errors.

       

      ========================================================================

      Starting PROD_0217@PROD Internal Concurrent Manager -- shell process ID 17553

       

       

       

      +---------------------------------------------------------------------------+

      Application Object Library: Concurrent Processing version 12.1

       

      Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.

       

      Internal Concurrent Manager started : 17-FEB-2016 14:08:13

       

      +---------------------------------------------------------------------------+

       

        Process monitor session started : 17-FEB-2016 14:08:13

      No such process

      An error occured in client-side routine afpsmckp for Service Manager FNDSM_ORAPRO_PROD. The routine returned code 1.

       

      Check for preceding errors and as well as the service manager log file for further details."

       

      17-FEB-2016 14:08:14 - Could not submit job to kill concurrent process 1707442: Oracle error 100: ORA-01403: no data found has been detected in FND_CONC_RAC_UTILS.SUBMIT_MANAGER_KILL_SESSION.

       

       

      Found dead process: spid=(6152), cpid=(1707442), ORA pid=(35), manager=(0/0)

      No such process

      An error occured in client-side routine afpsmckp for Service Manager FNDSM_ORAPRO_PROD. The routine returned code 1.

       

      Check for preceding errors and as well as the service manager log file for further details."

       

      17-FEB-2016 14:08:14 - Could not submit job to kill concurrent process 1707443: Oracle error 100: ORA-01403: no data found has been detected in FND_CONC_RAC_UTILS.SUBMIT_MANAGER_KILL_SESSION.

        logfile=/oracle/PROD/inst/apps/PROD_orapro/logs/appl/conc/log/PROD_0217.mgr  PRINTER=noprint  mailto=applprod  restart=N  diag=N  sleep=25  pmon=4  quesiz=1  Reviver is ENABLED

       

      But during this issue Standard Manager is also going down.

       

      Please suggest. Thanks in Advance,

      Karthik

        • 1. Re: R12.1.3 ICM going down - Actual Target mismatch
          VishnuVinnakota

          Give me the invalids of FND

          select object_name,object_type from dba_objects where status='INVALID' and object_name like 'FND%';

          What is your AD/TXK patch level and whats the latest ATG FP you applied?

          Did you migrate the appsutil.zip to DB tier and run Autoconfig?

          Run cmclean and fnd_conc_clone.setup_clean.

          Run Autoconfig on both DB and Apps Tier and try again.

          1 person found this helpful
          • 2. Re: R12.1.3 ICM going down - Actual Target mismatch
            Karthik Appsys

            Hi Vishnu,

             

            Thanks for your response.

            Give me the invalids of FND

            select object_name,object_type from dba_objects where status='INVALID' and object_name like 'FND%';

            SQL> select count(*) from all_objects where status='INVALID' and object_name like 'FND%';

              COUNT(*)

            ----------

                     0

            What is your AD/TXK patch level and whats the latest ATG FP you applied?

            12.1.3 RPC4

             

            Did you migrate the appsutil.zip to DB tier and run Autoconfig?

            Yes off course, after the latest RPC 4 patch as instructed in pre-requisite of RPC 4 we did it.

            Run cmclean and fnd_conc_clone.setup_clean.

            Run Autoconfig on both DB and Apps Tier and try again.

            Do you want me to perform in PROD ? If doing so will clear the FND_NODES values.

             

            Should I consider the following ?

            Routine FDPCRQ encountered an ORACLE error. ORA-04031: unable to

            allocate 40 bytes of shared memory ("shared pool".

            • 3. Re: R12.1.3 ICM going down - Actual Target mismatch
              VishnuVinnakota

              cmclean and fnd_conc_clone are not a big deal. However, as its PROD, you can opt for them at last.

               

              Give me output of :

               

              show parameter sga

              show parameter memory

              show parameter size

               

              Did you check below. Look like the ones.

               

              17089016   : KERNEL: FNDLIBR[30023]: SEGFAULT ERRORS IN OS LOGS

              17814970: Incorrect Error Handling for Terminated Requests.

               

              Post any error in Alert log as well.

               

              What the memory available on DB node? Give value from top or free commands.

              • 4. Re: R12.1.3 ICM going down - Actual Target mismatch
                Karthik Appsys

                Give me output of :

                 

                show parameter sga

                show parameter memory

                show parameter size

                SQL> show parameter sga

                 

                 

                NAME                                 TYPE        VALUE

                ------------------------------------ ----------- ------------------------------

                lock_sga                             boolean     FALSE

                pre_page_sga                         boolean     TRUE

                sga_max_size                         big integer 2G

                sga_target                           big integer 2G

                unified_audit_sga_queue_size         integer     1048576

                SQL> show parameter memory

                 

                 

                NAME                                 TYPE        VALUE

                ------------------------------------ ----------- ------------------------------

                hi_shared_memory_address             integer     0

                inmemory_clause_default              string

                inmemory_force                       string      DEFAULT

                inmemory_max_populate_servers        integer     0

                inmemory_query                       string      ENABLE

                inmemory_size                        big integer 0

                inmemory_trickle_repopulate_servers_ integer     1

                percent

                memory_max_target                    big integer 0

                memory_target                        big integer 0

                optimizer_inmemory_aware             boolean     TRUE

                 

                 

                NAME                                 TYPE        VALUE

                ------------------------------------ ----------- ------------------------------

                shared_memory_address                integer     0

                SQL> show parameter size

                 

                 

                NAME                                 TYPE        VALUE

                ------------------------------------ ----------- ------------------------------

                bitmap_merge_area_size               integer     1048576

                client_result_cache_size             big integer 0

                create_bitmap_area_size              integer     8388608

                db_16k_cache_size                    big integer 0

                db_2k_cache_size                     big integer 0

                db_32k_cache_size                    big integer 0

                db_4k_cache_size                     big integer 0

                db_8k_cache_size                     big integer 0

                db_block_size                        integer     8192

                db_cache_size                        big integer 0

                db_flash_cache_size                  big integer 0

                 

                 

                NAME                                 TYPE        VALUE

                ------------------------------------ ----------- ------------------------------

                db_keep_cache_size                   big integer 0

                db_recovery_file_dest_size           big integer 0

                db_recycle_cache_size                big integer 0

                dnfs_batch_size                      integer     4096

                global_context_pool_size             string

                hash_area_size                       integer     131072

                inmemory_size                        big integer 0

                java_max_sessionspace_size           integer     0

                java_pool_size                       big integer 0

                large_pool_size                      big integer 0

                max_dump_file_size                   string      UNLIMITED

                 

                 

                NAME                                 TYPE        VALUE

                ------------------------------------ ----------- ------------------------------

                max_string_size                      string      STANDARD

                object_cache_max_size_percent        integer     10

                object_cache_optimal_size            integer     102400

                olap_page_pool_size                  big integer 4M

                parallel_execution_message_size      integer     16384

                result_cache_max_size                big integer 8M

                sga_max_size                         big integer 2G

                shared_pool_reserved_size            big integer 95630131

                shared_pool_size                     big integer 800M

                sort_area_retained_size              integer     0

                sort_area_size                       integer     65536

                 

                 

                NAME                                 TYPE        VALUE

                ------------------------------------ ----------- ------------------------------

                streams_pool_size                    big integer 0

                unified_audit_sga_queue_size         integer     1048576

                workarea_size_policy                 string      AUTO

                 

                 

                Did you check below. Look like the ones.

                 

                17089016   : KERNEL: FNDLIBR[30023]: SEGFAULT ERRORS IN OS LOGS

                17814970: Incorrect Error Handling for Terminated Requests.

                 

                Post any error in Alert log as well.

                 

                What the memory available on DB node? Give value from top or free commands.

                No Errors in alert log. This is a Single Node EBS with 80 GB of RAM, 6 CPUs HP Proliant Gen 9 machine.

                • 5. Re: R12.1.3 ICM going down - Actual Target mismatch
                  VishnuVinnakota

                  SGA is undersized clearly. Incresee it to 8GB as you have 80GB anyways. Make use of in memory features. Use the memory advisor to estimate the best SGA/Memory Target for your env

                   

                  COming to current issue, did you check the documents I mentioned? Is this issue reproducible in test instance?

                  1 person found this helpful
                  • 6. Re: R12.1.3 ICM going down - Actual Target mismatch
                    Karthik Appsys

                    VishnuVinnakota wrote:

                     

                    SGA is undersized clearly. Incresee it to 8GB as you have 80GB anyways. Make use of in memory features. Use the memory advisor to estimate the best SGA/Memory Target for your env

                     

                    COming to current issue, did you check the documents I mentioned? Is this issue reproducible in test instance?

                    Let me increase the SGA and see how it behaves. Also our pga is


                    SQL> show parameter pga

                    NAME                                 TYPE        VALUE

                    ------------------------------------ ----------- ------------------------------

                    pga_aggregate_limit                  big integer 8G

                    pga_aggregate_target                 big integer 4G

                     

                    Yes I had a look at the docs. Both docs says the patch 17089016 which is also not applied in our instances

                     

                    but the version of the file "fdperi" in our instance is greater than what it is suggested in doc.

                    $ strings -a $FND_TOP/bin/FNDLIBR | grep Header | grep fdperi

                    $Header: fdperi.lc 120.0.12010000.4 2014/08/04 21:19:38 pferguso ship $

                     

                    Thanks,

                    Karthik

                    • 7. Re: R12.1.3 ICM going down - Actual Target mismatch
                      VishnuVinnakota

                      Did you manage to troubleshoot the issue?

                      • 8. Re: R12.1.3 ICM going down - Actual Target mismatch
                        Karthik Appsys

                        Hi Vishnu,

                         

                        I still don't get a idea to change the SGA. As you're telling the SGA is undersized, may I know how ?

                         

                        As per this doc "396009.1"

                         

                        I kept the SGA 2GB and PGA 4GB

                         

                        As you said If I go to 8GB for SGA then should I not consider to increase the PGA from 4GB ?

                         

                        Please advise.

                         

                        Regards,

                        Karthik