12 Replies Latest reply on Dec 7, 2015 7:07 PM by Hussein Sawwan-Oracle

    Concurrent manager goes down interpretently

    951027

      Hi All,

       

       

      We have a program that fires about 10,000 requests.

      we have defined a separate manager to process this requests.

      every 2-3 days this particular manager goes down.

      we have to deactivate and activate in order to make this up and running.

      EBS Version r12.1.3

      db: 11.2.0.3

      Below is the log at the time of issue, the last request processes was on 11-oct-2015 17:23

      after that the requests got in pending state

       

       

       

       

      AFPGMG--  11-OCT-2015 17:22:24

              Manager process: spid=(9821), cpid=(1772002), ORA pid=(260) manager=(20004/7428)

              Manager session alive, could not lock handle  10739106471073910647236, continue to next manager

       

       

       

       

       

       

       

       

      AFPGMG--  11-OCT-2015 17:22:24

              prev_num_running = 0, num_running = 191

       

       

       

       

       

       

       

       

                         Manager startup session started : 11-OCT-2015 17:22:24

                           Manager startup session ended : 11-OCT-2015 17:22:24

      11-OCT-2015 17:22:24 - monitor_self: icm_get_nodes executed:

      11-OCT-2015 17:22:24 -       target=TESTAP, pnode=TESTAP, pqueue=

      11-OCT-2015 17:22:24 -       snode=, squeue=

                           Process monitor session ended : 11-OCT-2015 17:22:24

       

       

       

       

      Sleeping for : 30 seconds

       

       

       

       

      Sleeping for : 30 seconds

       

       

       

       

      Sleeping for : 30 seconds

       

       

       

       

      Sleeping for : 30 seconds

       

       

       

       

                         Process monitor session started : 11-OCT-2015 17:24:25

                      Queue size posting session started : 11-OCT-2015 17:24:25

      Node list before Update

      Node list after Update

      Node=TESTAP, Inst=, SM Name=, Up=1

      Node=TESTDB, Inst=, SM Name=, Up=1

      Default node: TESTAP

       

       

       

       

      Any help will be highly appreciated.

      Thanks

        • 1. Re: Concurrent manager goes down interpretently
          Bashar.

          Hi,

           

          10,000 requests?!!

           

          What are the database configuration parameters?

          Is there anything in the database alert log?

           

          Regards,

          Bashar

          • 2. Re: Concurrent manager goes down interpretently
            951027

            Hi,

             

            Thanks for the reply bashar,

            There is no error in alert log which is observed.

            yes about 10,000 requests gets fired,

            Parameters are as follows

             

             

              

            NAMEVALUE
            lock_name_space
            processes1200
            sessions2400
            timed_statisticsTRUE
            timed_os_statistics0
            resource_limitFALSE
            license_max_sessions0
            license_sessions_warning0
            cpu_count12
            instance_groups
            event
            sga_max_size8589934592
            use_large_pagesTRUE
            pre_page_sgaFALSE
            shared_memory_address0
            hi_shared_memory_address0
            use_indirect_data_buffersFALSE
            lock_sgaFALSE
            processor_group_name
            shared_pool_size0
            large_pool_size0
            java_pool_size0
            streams_pool_size0
            shared_pool_reserved_size67108864
            java_soft_sessionspace_limit0
            java_max_sessionspace_size0
            spfile
            instance_typeRDBMS
            nls_languageAMERICAN
            nls_territoryAMERICA
            nls_sortBINARY
            nls_date_languageAMERICAN
            nls_date_formatDD-MON-RR
            nls_currency$
            nls_numeric_characters.,
            nls_iso_currencyAMERICA
            nls_calendarGREGORIAN
            nls_time_formatHH.MI.SSXFF AM
            nls_timestamp_formatDD-MON-RR HH.MI.SSXFF AM
            nls_time_tz_formatHH.MI.SSXFF AM TZR
            nls_timestamp_tz_formatDD-MON-RR HH.MI.SSXFF AM TZR
            nls_dual_currency$
            nls_compBINARY
            nls_length_semanticsBYTE
            nls_nchar_conv_excpFALSE
            fileio_network_adapters
            filesystemio_optionsnone
            clonedbFALSE
            disk_asynch_ioTRUE
            tape_asynch_ioTRUE
            dbwr_io_slaves0
            backup_tape_io_slavesFALSE
            resource_manager_cpu_allocation12
            resource_manager_planINTERNAL_PLAN
            cluster_interconnects
            file_mappingFALSE
            gcs_server_processes0
            active_instance_count
            sga_target8589934592
            memory_target0
            memory_max_target0
            control_files/d02/TESTCLN/db/apps_st/data/cntrl01.dbf, /d02/TESTCLN/db/apps_st/data/cntrl02.dbf, /d02/TESTCLN/db/apps_st/data/cntrl03.dbf
            db_file_name_convert
            log_file_name_convert
            control_file_record_keep_time7
            db_block_buffers0
            db_block_checksumTRUE
            db_ultra_safeOFF
            db_block_size8192
            db_cache_size0
            db_2k_cache_size0
            db_4k_cache_size0
            db_8k_cache_size0
            db_16k_cache_size0
            db_32k_cache_size0
            db_keep_cache_size0
            db_recycle_cache_size0
            db_writer_processes2
            buffer_pool_keep
            buffer_pool_recycle
            db_flash_cache_file
            db_flash_cache_size0
            db_cache_adviceON
            compatible11.2.0
            log_archive_dest_1LOCATION=/d02/TESTCLN/db/apps_st/data/archive
            log_archive_dest_2
            log_archive_dest_3
            log_archive_dest_4
            log_archive_dest_5
            log_archive_dest_6
            log_archive_dest_7
            log_archive_dest_8
            log_archive_dest_9
            log_archive_dest_10
            log_archive_dest_11
            log_archive_dest_12
            log_archive_dest_13
            log_archive_dest_14
            log_archive_dest_15
            log_archive_dest_16
            log_archive_dest_17
            log_archive_dest_18
            log_archive_dest_19
            log_archive_dest_20
            log_archive_dest_21
            log_archive_dest_22
            log_archive_dest_23
            log_archive_dest_24
            log_archive_dest_25
            log_archive_dest_26
            log_archive_dest_27
            log_archive_dest_28
            log_archive_dest_29
            log_archive_dest_30
            log_archive_dest_31
            log_archive_dest_state_1enable
            log_archive_dest_state_2enable
            log_archive_dest_state_3enable
            log_archive_dest_state_4enable
            log_archive_dest_state_5enable
            log_archive_dest_state_6enable
            log_archive_dest_state_7enable
            log_archive_dest_state_8enable
            log_archive_dest_state_9enable
            log_archive_dest_state_10enable
            log_archive_dest_state_11enable
            log_archive_dest_state_12enable
            log_archive_dest_state_13enable
            log_archive_dest_state_14enable
            log_archive_dest_state_15enable
            log_archive_dest_state_16enable
            log_archive_dest_state_17enable
            log_archive_dest_state_18enable
            log_archive_dest_state_19enable
            log_archive_dest_state_20enable
            log_archive_dest_state_21enable
            log_archive_dest_state_22enable
            log_archive_dest_state_23enable
            log_archive_dest_state_24enable
            log_archive_dest_state_25enable
            log_archive_dest_state_26enable
            log_archive_dest_state_27enable
            log_archive_dest_state_28enable
            log_archive_dest_state_29enable
            log_archive_dest_state_30enable
            log_archive_dest_state_31enable
            log_archive_startFALSE
            log_archive_dest
            log_archive_duplex_dest
            log_archive_min_succeed_dest1
            standby_archive_dest?/dbs/arch
            fal_client
            fal_server
            log_archive_trace0
            log_archive_config
            log_archive_local_firstTRUE
            log_archive_format%t_%s_%r.dbf
            redo_transport_user
            log_archive_max_processes4
            log_buffer10485760
            log_checkpoint_interval100000
            log_checkpoint_timeout1200
            archive_lag_target0
            db_files1024
            db_file_multiblock_read_count128
            read_only_open_delayedFALSE
            cluster_databaseFALSE
            parallel_serverFALSE
            parallel_server_instances1
            cluster_database_instances1
            db_create_file_dest
            db_create_online_log_dest_1
            db_create_online_log_dest_2
            db_create_online_log_dest_3
            db_create_online_log_dest_4
            db_create_online_log_dest_5
            db_recovery_file_dest
            db_recovery_file_dest_size0
            standby_file_managementMANUAL
            db_unrecoverable_scn_trackingTRUE
            thread0
            fast_start_io_target0
            fast_start_mttr_target0
            log_checkpoints_to_alertTRUE
            db_lost_write_protectNONE
            recovery_parallelism0
            db_flashback_retention_target1440
            dml_locks10000
            replication_dependency_trackingTRUE
            transactions2640
            transactions_per_rollback_segment5
            rollback_segments
            undo_managementAUTO
            undo_tablespaceAPPS_UNDOTS1
            undo_retention900
            fast_start_parallel_rollbackLOW
            resumable_timeout0
            instance_number0
            db_block_checkingFALSE
            recyclebinon
            db_securefilePERMITTED
            create_stored_outlines
            serial_reusedisable
            ldap_directory_accessNONE
            ldap_directory_sysauthno
            os_rolesFALSE
            rdbms_server_dn
            max_enabled_roles150
            remote_os_authentFALSE
            remote_os_rolesFALSE
            sec_case_sensitive_logonFALSE
            O7_DICTIONARY_ACCESSIBILITYFALSE
            remote_login_passwordfileEXCLUSIVE
            license_max_users0
            audit_sys_operationsTRUE
            global_context_pool_size
            db_domain
            global_namesFALSE
            distributed_lock_timeout60
            commit_point_strength1
            global_txn_processes1
            instance_nameTESTCLN
            service_namesTESTCLN
            dispatchers
            shared_servers0
            max_shared_servers
            max_dispatchers
            circuits
            shared_server_sessions
            local_listenerTESTCLN_LOCAL
            remote_listener
            listener_networks
            cursor_space_for_timeFALSE
            session_cached_cursors2500
            remote_dependencies_modeTIMESTAMP
            utl_file_dir/usr/tmp, /usr/tmp, /d02/TESTCLNdb/db/tech_st/11.2.0/appsutil/outbound/TESTCLN_dxbhoebtstdb, /usr/tmp
            smtp_out_server
            plsql_v2_compatibilityFALSE
            plsql_warningsDISABLE:ALL
            plsql_code_typeINTERPRETED
            plsql_debugFALSE
            plsql_optimize_level2
            plsql_ccflags
            plscope_settingsIDENTIFIERS:NONE
            permit_92_wrap_formatTRUE
            java_jit_enabledTRUE
            job_queue_processes2
            parallel_min_percent0
            _system_trig_enabledTRUE
            create_bitmap_area_size8388608
            bitmap_merge_area_size1048576
            cursor_sharingEXACT
            result_cache_modeMANUAL
            parallel_min_servers0
            parallel_max_servers8
            parallel_instance_group
            parallel_execution_message_size16384
            hash_area_size131072
            result_cache_max_size42958848
            result_cache_max_result5
            result_cache_remote_expiration0
            shadow_core_dumppartial
            background_core_dumppartial
            background_dump_dest/d02/TESTCLNdb/db/tech_st/11.2.0/admin/TESTCLN_dxbhoebtstdb/diag/rdbms/TESTCLN/TESTCLN/trace
            user_dump_dest/d02/TESTCLNdb/db/tech_st/11.2.0/admin/TESTCLN_dxbhoebtstdb/diag/rdbms/TESTCLN/TESTCLN/trace
            core_dump_dest/d02/TESTCLNdb/db/tech_st/11.2.0/admin/TESTCLN_dxbhoebtstdb/diag/rdbms/TESTCLN/TESTCLN/cdump
            audit_file_dest/d02/TESTCLNdb/db/tech_st/11.2.0/rdbms/audit
            audit_syslog_level
            object_cache_optimal_size102400
            object_cache_max_size_percent10
            session_max_open_files10
            open_links4
            open_links_per_instance4
            commit_write
            commit_wait
            commit_logging
            optimizer_features_enable11.2.0.3
            fixed_date
            audit_trailNONE
            sort_area_size65536
            sort_area_retained_size0
            cell_offload_processingTRUE
            cell_offload_decryptionTRUE
            cell_offload_parameters
            cell_offload_compactionADAPTIVE
            cell_offload_plan_displayAUTO
            db_nameTESTCLN
            db_unique_nameTESTCLN
            open_cursors1600
            ifile/d02/TESTCLNdb/db/tech_st/11.2.0/dbs/TESTCLN_dxbhoebtstdb_ifile.ora
            sql_traceFALSE
            os_authent_prefix
            optimizer_modeALL_ROWS
            _sort_elimination_cost_ratio5
            sql92_securityTRUE
            blank_trimmingFALSE
            _b_tree_bitmap_plansFALSE
            star_transformation_enabledFALSE
            _fast_full_scan_enabledFALSE
            parallel_degree_policyMANUAL
            parallel_adaptive_multi_userTRUE
            parallel_threads_per_cpu2
            parallel_automatic_tuningFALSE
            parallel_io_cap_enabledFALSE
            optimizer_index_cost_adj100
            optimizer_index_caching0
            query_rewrite_enabledTRUE
            query_rewrite_integrityenforced
            _like_with_bind_as_equalityTRUE
            pga_aggregate_target3221225472
            workarea_size_policyAUTO
            optimizer_dynamic_sampling2
            _optimizer_autostats_jobFALSE
            statistics_levelTYPICAL
            cursor_bind_capture_destinationmemory+disk
            skip_unusable_indexesTRUE
            optimizer_secure_view_mergingFALSE
            ddl_lock_timeout0
            deferred_segment_creationTRUE
            optimizer_use_pending_statisticsFALSE
            optimizer_capture_sql_plan_baselinesFALSE
            optimizer_use_sql_plan_baselinesTRUE
            parallel_min_time_thresholdAUTO
            parallel_degree_limitCPU
            parallel_force_localFALSE
            optimizer_use_invisible_indexesFALSE
            dst_upgrade_insert_convTRUE
            parallel_servers_target8
            sec_protocol_error_trace_actionTRACE
            sec_protocol_error_further_actionCONTINUE
            sec_max_failed_login_attempts10
            sec_return_server_release_bannerFALSE
            enable_ddl_loggingFALSE
            client_result_cache_size0
            client_result_cache_lag3000
            aq_tm_processes1
            hs_autoregisterTRUE
            xml_db_eventsenable
            dg_broker_startFALSE
            dg_broker_config_file1/d02/TESTCLNdb/db/tech_st/11.2.0/dbs/dr1TESTCLN.dat
            dg_broker_config_file2/d02/TESTCLNdb/db/tech_st/11.2.0/dbs/dr2TESTCLN.dat
            olap_page_pool_size4194304
            asm_diskstring
            asm_preferred_read_failure_groups
            asm_diskgroups
            asm_power_limit1
            control_management_pack_accessDIAGNOSTIC+TUNING
            awr_snapshot_time_offset0
            sqltune_categoryDEFAULT
            diagnostic_dest/d02/TESTCLNdb/db/tech_st/11.2.0/admin/TESTCLN_dxbhoebtstdb
            tracefile_identifier
            _trace_files_publicFALSE
            max_dump_file_size20480
            trace_enabledTRUE
            • 3. Re: Concurrent manager goes down interpretently
              951027

              Hello all,

               

              Please check if you may find anything on this.

               

              Thanks

              • 4. Re: Concurrent manager goes down interpretently
                951027

                I am not able to find any particular links related to this issue.

                • 5. Re: Concurrent manager goes down interpretently
                  951027

                  Hi Bahsar,

                   

                  let me know if you need any further inputs.

                  • 6. Re: Concurrent manager goes down interpretently
                    Bashar.

                    There is a problem with the Inbox, I did not notice your previous updates.

                     

                    Does the manager go down while processing the requests or while it is idle?

                    Did you search the log for any other errors or messages?

                     

                    Regards,

                    Bashar

                    • 7. Re: Concurrent manager goes down interpretently
                      951027

                      Sorry for the delay, Yes it gets stopped while processing.

                      • 8. Re: Concurrent manager goes down interpretently
                        951027

                        Hi All,

                         

                        Any one facing the same issue?

                        • 9. Re: Concurrent manager goes down interpretently
                          Jignesh Makwana

                          Hello

                           

                          Sleeping for : 30 seconds

                           

                          I suggest to change this sleep time from 30 to 10 and see the result.

                           

                          it help me one time when I was heaving same issue.

                           

                          Jignesh

                          • 10. Re: Concurrent manager goes down interpretently
                            951027

                            Hi Jignesh,

                             

                            I tried that as well but stil after running for some time its getting stopped

                            • 11. Re: Concurrent manager goes down interpretently
                              951027

                              Hi All,

                               

                              Any help on this, anyone facing similar issue?

                               

                              Thanks

                              • 12. Re: Concurrent manager goes down interpretently
                                Hussein Sawwan-Oracle

                                Hi All,

                                 

                                Any help on this, anyone facing similar issue?

                                 

                                Thanks

                                 

                                How do you fix the issue? Do you just bounce the CM?

                                 

                                Have you tried to run (Concurrent Processing - Concurrent Manager Recovery Troubleshooting Wizard (Doc ID 134007.1)) and see if it helps?

                                 

                                Try to start the CM with "diag=y" and see if more details are collected in the logs.

                                 

                                I'd also suggest you log a SR.

                                 

                                Thanks,

                                Hussein