1 2 Previous Next 19 Replies Latest reply: Apr 25, 2014 11:31 AM by yoonas RSS

    Heartbeat failed to connect to standby . Error is 16058

    User348341

      Hello all, I have been trying to configure data guard recently on oracle 10gr2 but I am having error in my physical standby database as shown below. "*** 2014-04-22 10:08:07.581 68056 kcrr.c Logged on to standby successfully Client logon and security negotiation successful! Error 16058 attaching RFS server to standby instance at host 'p_stdy' Error 16058 attaching to destination LOG_ARCHIVE_DEST_2 standby host 'p_stdy' ORA-16058: Message 16058 not found; No message file for product=RDBMS, facility=ORA *** 2014-04-22 10:08:07.587 64213 kcrr.c PING[ARC1]: Heartbeat failed to connect to standby 'p_stdy'. Error is 16058. *** 2014-04-22 10:08:07.587 62481 kcrr.c kcrrfail: dest:2 err:16058 force:0 blast:1 *** 2014-04-22 10:09:07.600 Redo shipping client performing standby login *** 2014-04-22 10:09:27.661 *** 2014-04-22 10:09:27.661 68056 kcrr.c Logged on to standby successfully Client logon and security negotiation successful! Error 16058 attaching RFS server to standby instance at host 'p_stdy' Error 16058 attaching to destination LOG_ARCHIVE_DEST_2 standby host 'p_stdy' ORA-16058: Message 16058 not found; No message file for product=RDBMS, facility=ORA" Pls can anyone help.

        • 1. Re: Heartbeat failed to connect to standby . Error is 16058
          User348341

          I did a tnsping to standby db and it was successful also trying it vice versa was successful as well. Now not sure what this might be.

          • 2. Re: Heartbeat failed to connect to standby . Error is 16058
            Renu-Oracle

            Hi,

             

            Provide below:

             

            SQL>SHOW PARAMETER LOG_ARCHIVE_DEST_2;

             

            What are the errors reported in alert.log?

             

            Thanks,

            Renu

            • 3. Re: Heartbeat failed to connect to standby . Error is 16058
              yoonas

              Hi,

               

              On Primary check

               

              SELECT des_name,
                status,
                type,
                database_mode,
                recovery_mode,
                error,
                destination,
                SYNCHRONIZATION_STATUS
              FROM V$ARCHIVE_DEST_STATUS
              

               

               

              What are  the errors from primary/standby log files apart from above posted?

              Just to make sure, the above errors are from standby alert log?

               

              Regards

              Yoonas

              • 4. Re: Heartbeat failed to connect to standby . Error is 16058
                User348341
                > select dest_name, status, type, database_mode, recovery_mode, error, destination, synchronization_status
                from v$archive_dest_status
                DEST_NAME                     STATUS    TYPE           DATABASE_MODE   RECOVERY_MODE           ERROR                                                    
                --------------------------------------------------------------------------------------------------------------------------------------------------------
                LOG_ARCHIVE_DEST_1          VALID     LOCAL          OPEN            IDLE                                                                               
                LOG_ARCHIVE_DEST_2          ERROR     PHYSICAL       UNKNOWN         UNKNOWN                 ORA-16058: standby database instance is not mounted        
                LOG_ARCHIVE_DEST_3          INACTIVE  LOCAL          OPEN            IDLE                                                                               
                LOG_ARCHIVE_DEST_4          INACTIVE  LOCAL          OPEN            IDLE                                                                               
                LOG_ARCHIVE_DEST_5          INACTIVE  LOCAL          OPEN            IDLE                                                                               
                LOG_ARCHIVE_DEST_6          INACTIVE  LOCAL          OPEN            IDLE                                                                               
                LOG_ARCHIVE_DEST_7          INACTIVE  LOCAL          OPEN            IDLE                                                                               
                LOG_ARCHIVE_DEST_8          INACTIVE  LOCAL          OPEN            IDLE                                                                               
                LOG_ARCHIVE_DEST_9          INACTIVE  LOCAL          OPEN            IDLE                                                                               
                LOG_ARCHIVE_DEST_10         INACTIVE  LOCAL          OPEN            IDLE                                                                               
                
                
                 10 rows selected 
                

                 

                For Physical  Alert Log

                Wed Apr 23 01:48:08 2014
                PING[ARC1]: Heartbeat failed to connect to standby 'p_stdy'. Error is 16058.
                Wed Apr 23 01:49:28 2014
                Errors in file /pcard18/admin/pcard/bdump/pcard_arc1_23200.trc:
                ORA-16058: Message 16058 not found; No message file for product=RDBMS, facility=ORA
                Wed Apr 23 01:49:28 2014
                PING[ARC1]: Heartbeat failed to connect to standby 'p_stdy'. Error is 16058.
                Wed Apr 23 01:50:48 2014
                Errors in file /pcard18/admin/pcard/bdump/pcard_arc1_23200.trc:
                ORA-16058: Message 16058 not found; No message file for product=RDBMS, facility=ORA
                Wed Apr 23 01:50:48 2014
                PING[ARC1]: Heartbeat failed to connect to standby 'p_stdy'. Error is 16058.
                Wed Apr 23 01:52:08 2014
                Errors in file /pcard18/admin/pcard/bdump/pcard_arc1_23200.trc:
                ORA-16058: Message 16058 not found; No message file for product=RDBMS, facility=ORA
                Wed Apr 23 01:52:08 2014
                PING[ARC1]: Heartbeat failed to connect to standby 'p_stdy'. Error is 16058.
                Wed Apr 23 01:53:28 2014
                Errors in file /pcard18/admin/pcard/bdump/pcard_arc1_23200.trc:
                ORA-16058: Message 16058 not found; No message file for product=RDBMS, facility=ORA
                Wed Apr 23 01:53:28 2014
                PING[ARC1]: Heartbeat failed to connect to standby 'p_stdy'. Error is 16058.
                Wed Apr 23 01:54:48 2014
                Errors in file /pcard18/admin/pcard/bdump/pcard_arc1_23200.trc:
                ORA-16058: Message 16058 not found; No message file for product=RDBMS, facility=ORA
                Wed Apr 23 01:54:48 2014
                PING[ARC1]: Heartbeat failed to connect to standby 'p_stdy'. Error is 16058.
                Wed Apr 23 01:56:08 2014
                Errors in file /pcard18/admin/pcard/bdump/pcard_arc1_23200.trc:
                ORA-16058: Message 16058 not found; No message file for product=RDBMS, facility=ORA
                Wed Apr 23 01:56:08 2014
                PING[ARC1]: Heartbeat failed to connect to standby 'p_stdy'. Error is 16058.
                Wed Apr 23 01:57:28 2014
                Errors in file /pcard18/admin/pcard/bdump/pcard_arc1_23200.trc:
                ORA-16058: Message 16058 not found; No message file for product=RDBMS, facility=ORA
                Wed Apr 23 01:57:28 2014
                PING[ARC1]: Heartbeat failed to connect to standby 'p_stdy'. Error is 16058.
                

                 

                For Standby alert Log

                ARC0: Thread not mounted
                Wed Apr 23 02:46:37 2014
                ARC0: Archival started
                ARC1: Archival started
                ARCH: STARTING ARCH PROCESSES COMPLETE
                ARC1 started with pid=16, OS id=20611
                ARC1: Becoming the 'no FAL' ARCH
                ARC1: Becoming the 'no SRL' ARCH
                ARC1: Thread not mounted
                Wed Apr 23 02:46:37 2014
                ARC0: Becoming the heartbeat ARCH
                ARC0: Thread not mounted
                Wed Apr 23 02:46:37 2014
                Successful mount of redo thread 1, with mount id 1942641013
                Wed Apr 23 02:46:37 2014
                Physical Standby Database mounted.
                Completed: ALTER DATABASE   MOUNT
                Wed Apr 23 02:46:41 2014
                Starting Data Guard Broker (DMON)
                INSV started with pid=17, OS id=20616
                Wed Apr 23 02:47:50 2014
                alter database recover managed standby database disconnect
                Wed Apr 23 02:47:50 2014
                Attempt to start background Managed Standby Recovery process (P_STDY)
                MRP0 started with pid=18, OS id=20619
                Wed Apr 23 02:48:00 2014
                MRP0: Background Managed Standby Recovery process started (P_STDY)
                Managed Standby Recovery not using Real Time Apply
                Wed Apr 23 02:48:25 2014
                 parallel recovery started with 2 processes
                Wed Apr 23 02:48:25 2014
                Waiting for all non-current ORLs to be archived...
                Media Recovery Waiting for thread 1 sequence 3069
                Wed Apr 23 02:48:25 2014
                Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_20619.trc:
                ORA-01017: Message 1017 not found; No message file for product=RDBMS, facility=ORA
                Error 1017 received logging on to the standby
                ------------------------------------------------------------
                Check that the primary and standby are using a password file
                and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
                and that the SYS password is same in the password files.
                      returning error ORA-16191
                ------------------------------------------------------------
                FAL[client, MRP0]: Error 16191 connecting to PCARD for fetching gap sequence
                Wed Apr 23 02:48:25 2014
                Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_20619.trc:
                ORA-16191: Message 16191 not found; No message file for product=RDBMS, facility=ORA
                Wed Apr 23 02:48:26 2014
                Completed: alter database recover managed standby database disconnect
                
                
                • 5. Re: Heartbeat failed to connect to standby . Error is 16058
                  yoonas

                  Hi,

                   

                   

                  Check that the primary and standby are using a password file

                  and remote_login_passwordfile is set to SHARED or EXCLUSIVE,

                  and that the SYS password is same in the password files.

                        returning error ORA-16191

                  Please make sure password file is created as said above

                   

                  Are you able to connect to standby with service_name (p_std)

                   

                  Try

                  sys@service_name as sysdba
                  

                   

                  Regards

                  Yoonas

                  • 6. Re: Heartbeat failed to connect to standby . Error is 16058
                    User348341

                    Thanks Yoonas, Yes they have the same password file. In fact i can connect to standby DB from Primary

                     

                    SQL> conn sys@p_stdy as sysdba
                    Enter password:
                    Connected.
                    SQL> show parameter db_unique_name
                    
                    NAME                                 TYPE       VALUE
                    ------------------------------------ ---------- ------------------------------
                    db_unique_name                       string     p_stdy
                    
                    
                    

                     

                     

                    While in Primary DB (pcard), i can connect to standby which is a sign that my passwordfile are the same.

                     

                    I will also like to point out that the two databases have the same name (pcard)

                    • 7. Re: Heartbeat failed to connect to standby . Error is 16058
                      User348341

                      I will also like to add that anytime i connect to standby from primary, after a while i do have "end of file communication channel". Please could this be an issue

                      Please see below

                       

                      SQL> conn sys@p_stdy as sysdba
                      Enter password:
                      Connected.
                      SQL> show parameter db_unique_name
                      
                      
                      NAME                                 TYPE       VALUE
                      ------------------------------------ ---------- ------------------------------
                      db_unique_name                       string     p_stdy
                      SQL> show paramaeter db_name
                      
                      
                      SQL> shpw parameter db_name
                      SP2-0734: unknown command beginning "shpw param..." - rest of line ignored.
                      SQL> show parameter db_name
                      ORA-03113: end-of-file on communication channel
                      
                      
                      
                      
                      SQL> show parameter db_unique_name
                      ERROR:
                      ORA-03114: not connected to ORACLE
                      
                      
                      
                      
                      SQL> select instance_name from v$instance;
                      ERROR:
                      ORA-03114: not connected to ORACLE
                      
                      
                      
                      
                      SQL> disconnect
                      Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
                      With the Partitioning, OLAP, Data Mining and Real Application Testing options
                      SQL>
                      SQL>
                      SQL> conn / as sysdba
                      Connected.
                      SQL>
                      SQL>
                      SQL> show parameter db_name
                      
                      
                      NAME                                 TYPE       VALUE
                      ------------------------------------ ---------- ------------------------------
                      db_name                              string     pcard
                      SQL>
                      SQL>
                      SQL> conn sys@p_stdy as sysdba
                      Enter password:
                      Connected.
                      SQL> show parameter db_name
                      
                      
                      NAME                                 TYPE       VALUE
                      ------------------------------------ ---------- ------------------------------
                      db_name                              string     p_stdy
                      SQL> show parameter  remote_login_passwordfile
                      
                      
                      SQL> show parameter  remote_login_passwordfile
                      ORA-03113: end-of-file on communication channel
                      
                      
                      
                      
                      SQL> disconnect
                      Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
                      With the Partitioning, OLAP, Data Mining and Real Application Testing options
                      SQL> conn / as sysdba
                      Connected.
                      SQL> show parameter  remote_login_passwordfile
                      
                      
                      NAME                                 TYPE       VALUE
                      ------------------------------------ ---------- ------------------------------
                      remote_login_passwordfile            string     EXCLUSIVE
                      SQL>
                      SQL> conn sys@p_stdy as sysdba
                      Enter password:
                      Connected.
                      SQL> show parameter  remote_login_passwordfile
                      
                      
                      NAME                                 TYPE       VALUE
                      ------------------------------------ ---------- ------------------------------
                      remote_login_passwordfile            string     EXCLUSIVE
                      SQL>
                      
                      
                      • 8. Re: Heartbeat failed to connect to standby . Error is 16058
                        BPeaslandDBA

                        [oracle@nau-rac01 ~]$ oerr ora 16058

                        16058, 00000, "standby database instance is not mounted"

                        // *Cause:  The Remote File Server (RFS) process on the standby database

                        //          received an internal error.

                        // *Action: Check the standby alert log and RFS trace files for more

                        //          information.

                         

                        Are you sure that your standby instance is MOUNTed? It must be in MOUNT mode before the primary can send redo to the standby.

                         

                        And it is possible that tnsping works just fine with the instance not mounted, depending on your configuration. So that is not an accurate test. Are more accurate test would be if you could do this on the primary:

                         

                        sqlplus sys/password@standby as sysdba

                         

                         

                        HTH,

                        Brian

                        • 9. Re: Heartbeat failed to connect to standby . Error is 16058
                          User348341

                          Thanks Brian,

                           

                          I found something strange to me. from primary DB i was told DB not mounted but when i logged on to the standby DB it was actually mounted. please find below:

                           

                          From Primary DB I launched Sqlplus to connect to the Standby as you advised

                           

                          computer_name:/oracle/home>sqlplus sys@p_stdy as sysdba
                          
                          
                          SQL*Plus: Release 10.2.0.4.0 - Production on Wed Apr 23 05:53:52 2014
                          
                          
                          Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.
                          
                          
                          Enter password:
                          
                          
                          Connected to:
                          Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
                          With the Partitioning, OLAP, Data Mining and Real Application Testing options
                          
                          
                          SQL> select instance_name from v$instance;
                          
                          
                          INSTANCE_NAME
                          ----------------
                          p_stdy
                          
                          
                          SQL> select database_role from v$database;
                          select database_role from v$database
                                                    *
                          ERROR at line 1:
                          ORA-01507: database not mounted
                          
                          
                          

                           

                          But when I logged on to the Standby database directly it was mounted

                           

                          SQL> select status from v$instance;
                          
                          STATUS
                          ------------
                          MOUNTED
                          
                          
                          SQL>
                          
                          

                          Please help.

                          • 10. Re: Heartbeat failed to connect to standby . Error is 16058
                            User348341

                            Howerver, i restarted Standby DB to mount mode but still have similar error.

                            Please find below for the alert log content

                             

                            Shutting down instance (immediate)
                            License high water mark = 3
                            Wed Apr 23 06:11:27 2014
                            Stopping Job queue slave processes, flags = 7
                            Wed Apr 23 06:11:27 2014
                            Job queue slave processes stopped
                            Wed Apr 23 06:11:27 2014
                            ALTER DATABASE CLOSE NORMAL
                            ORA-1109 signalled during: ALTER DATABASE CLOSE NORMAL...
                            Wed Apr 23 06:11:27 2014
                            ALTER DATABASE DISMOUNT
                            Completed: ALTER DATABASE DISMOUNT
                            ARCH: Archival disabled due to shutdown: 1089
                            Shutting down archive processes
                            Archiving is disabled
                            Wed Apr 23 06:11:31 2014
                            ARC0: Archival disabled due to shutdown: 1089
                            Shutting down archive processes
                            Wed Apr 23 06:11:32 2014
                            ARCH shutting down
                            ARC1: Archival stopped
                            Wed Apr 23 06:11:37 2014
                            Archiving is disabled
                            Archive process shutdown avoided: 0 active
                            ARCH shutting down
                            ARC0: Archival stopped
                            Wed Apr 23 06:11:38 2014
                            Shutting down Data Guard Broker processes
                            Wed Apr 23 06:11:39 2014
                            Completed: Data Guard Broker shutdown
                            Wed Apr 23 06:11:41 2014
                            ARCH: Archival disabled due to shutdown: 1089
                            Shutting down archive processes
                            Archiving is disabled
                            Archive process shutdown avoided: 0 active
                            Wed Apr 23 06:11:52 2014
                            Starting ORACLE instance (normal)
                            LICENSE_MAX_SESSION = 0
                            LICENSE_SESSIONS_WARNING = 0
                            Picked latch-free SCN scheme 2
                            Autotune of undo retention is turned on.
                            IMODE=BR
                            ILAT =24
                            LICENSE_MAX_USERS = 0
                            SYS auditing is disabled
                            Wed Apr 23 06:12:02 2014
                            ksdpec: called for event 13740 prior to event group initialization
                            Starting up ORACLE RDBMS Version: 10.2.0.4.0.
                            System parameters with non-default values:
                              processes                = 200
                              shared_pool_size         = 318767104
                              large_pool_size          = 0
                              java_pool_size           = 33554432
                              streams_pool_size        = 33554432
                              nls_language             = FRENCH
                              nls_territory            = FRANCE
                              sga_target               = 0
                              control_files            = /pcard02/oradata/p_stdy/control01.ctl, /pcard03/oradata/p_stdy/control02.ctl, /pcard04/oradata/p_stdy/control03.ctl
                              db_file_name_convert     = /oradata/pcard, /oradata/p_stdy
                              log_file_name_convert    = /oradata/pcard, /oradata/p_stdy
                              db_block_size            = 8192
                              db_cache_size            = 805306368
                              compatible               = 10.2.0.1.0
                              log_archive_config       = dg_config=(pcard,p_stdy)
                              log_archive_dest_1       = location=/pcard15/oradata/p_stdy/arch  valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=p_stdy
                              log_archive_dest_2       = service=pcard LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=pcard
                              log_archive_dest_state_1 = ENABLE
                              log_archive_dest_state_2 = ENABLE
                              log_archive_max_processes= 2
                              log_archive_min_succeed_dest= 1
                              log_archive_format       = %t_%s_%r.dbf
                              fal_client               = P_STDY
                              fal_server               = PCARD
                              archive_lag_target       = 0
                              db_file_multiblock_read_count= 16
                              db_recovery_file_dest    = /ora_backup
                              db_recovery_file_dest_size= 198642237440
                              standby_file_management  = AUTO
                              undo_management          = AUTO
                              undo_tablespace          = UNDOTBS1
                              remote_login_passwordfile= EXCLUSIVE
                              db_domain                =
                              job_queue_processes      = 10
                              background_dump_dest     = /pcard18/admin/p_stdy/bdump
                              user_dump_dest           = /pcard18/admin/p_stdy/udump
                              core_dump_dest           = /pcard18/admin/p_stdy/cdump
                              audit_file_dest          = /pcard18/admin/p_stdy/adump
                              audit_trail              = OS
                              db_name                  = pcard
                              db_unique_name           = p_stdy
                              open_cursors             = 300
                              pga_aggregate_target     = 209715200
                              dg_broker_start          = TRUE
                              _awr_flush_threshold_metrics= TRUE
                            PMON started with pid=2, OS id=24007
                            PSP0 started with pid=3, OS id=24009
                            MMAN started with pid=4, OS id=24011
                            Wed Apr 23 06:12:52 2014
                            ORA-00130: Message 130 not found; No message file for product=RDBMS, facility=ORA; arguments: [(ADDRESS=(PROTOCOL=TCP)(HOST=pwcd-stage-dr)(PORT=1521))]
                            DBW0 started with pid=5, OS id=24013
                            LGWR started with pid=6, OS id=24017
                            CKPT started with pid=7, OS id=24019
                            SMON started with pid=8, OS id=24021
                            RECO started with pid=9, OS id=24023
                            CJQ0 started with pid=10, OS id=24025
                            MMON started with pid=11, OS id=24027
                            MMNL started with pid=12, OS id=24029
                            DMON started with pid=13, OS id=24031
                            Wed Apr 23 06:14:12 2014
                            ALTER DATABASE   MOUNT
                            Wed Apr 23 06:14:16 2014
                            Setting recovery target incarnation to 1
                            ARCH: STARTING ARCH PROCESSES
                            ARC0 started with pid=15, OS id=24039
                            ARC0: Becoming the 'no FAL' ARCH
                            ARC0: Becoming the 'no SRL' ARCH
                            ARC0: Thread not mounted
                            ARC1 started with pid=16, OS id=24041
                            Wed Apr 23 06:14:36 2014
                            ARC0: Archival started
                            ARC1: Archival started
                            ARCH: STARTING ARCH PROCESSES COMPLETE
                            Wed Apr 23 06:14:36 2014
                            ARC0: Becoming the 'no FAL' ARCH
                            ARC0: Becoming the 'no SRL' ARCH
                            ARC0: Thread not mounted
                            Wed Apr 23 06:14:36 2014
                            ARC1: Becoming the heartbeat ARCH
                            ARC1: Thread not mounted
                            Wed Apr 23 06:14:36 2014
                            Successful mount of redo thread 1, with mount id 1942708020
                            Wed Apr 23 06:14:36 2014
                            Physical Standby Database mounted.
                            Completed: ALTER DATABASE   MOUNT
                            Wed Apr 23 06:14:40 2014
                            Starting Data Guard Broker (DMON)
                            INSV started with pid=17, OS id=24044
                            Wed Apr 23 06:15:16 2014
                            alter database recover managed standby database disconnect
                            Wed Apr 23 06:15:16 2014
                            Attempt to start background Managed Standby Recovery process (P_STDY)
                            MRP0 started with pid=18, OS id=24046
                            Wed Apr 23 06:15:26 2014
                            MRP0: Background Managed Standby Recovery process started (P_STDY)
                            Managed Standby Recovery not using Real Time Apply
                            Wed Apr 23 06:15:51 2014
                             parallel recovery started with 2 processes
                            Wed Apr 23 06:15:51 2014
                            Waiting for all non-current ORLs to be archived...
                            Media Recovery Waiting for thread 1 sequence 3069
                            Wed Apr 23 06:15:51 2014
                            Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_24046.trc:
                            ORA-01017: Message 1017 not found; No message file for product=RDBMS, facility=ORA
                            Error 1017 received logging on to the standby
                            ------------------------------------------------------------
                            Check that the primary and standby are using a password file
                            and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
                            and that the SYS password is same in the password files.
                                  returning error ORA-16191
                            ------------------------------------------------------------
                            FAL[client, MRP0]: Error 16191 connecting to PCARD for fetching gap sequence
                            Wed Apr 23 06:15:51 2014
                            Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_24046.trc:
                            ORA-16191: Message 16191 not found; No message file for product=RDBMS, facility=ORA
                            Wed Apr 23 06:15:52 2014
                            Completed: alter database recover managed standby database disconnect
                            Wed Apr 23 06:16:21 2014
                            Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_24046.trc:
                            ORA-01017: Message 1017 not found; No message file for product=RDBMS, facility=ORA
                            Wed Apr 23 06:16:21 2014
                            Error 1017 received logging on to the standby
                            ------------------------------------------------------------
                            Check that the primary and standby are using a password file
                            and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
                            and that the SYS password is same in the password files.
                                  returning error ORA-16191
                            ------------------------------------------------------------
                            FAL[client, MRP0]: Error 16191 connecting to PCARD for fetching gap sequence
                            Wed Apr 23 06:16:21 2014
                            Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_24046.trc:
                            ORA-16191: Message 16191 not found; No message file for product=RDBMS, facility=ORA
                            Wed Apr 23 06:16:51 2014
                            Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_24046.trc:
                            ORA-01017: Message 1017 not found; No message file for product=RDBMS, facility=ORA
                            Wed Apr 23 06:16:51 2014
                            Error 1017 received logging on to the standby
                            ------------------------------------------------------------
                            Check that the primary and standby are using a password file
                            and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
                            and that the SYS password is same in the password files.
                                  returning error ORA-16191
                            ------------------------------------------------------------
                            FAL[client, MRP0]: Error 16191 connecting to PCARD for fetching gap sequence
                            Wed Apr 23 06:16:51 2014
                            Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_24046.trc:
                            ORA-16191: Message 16191 not found; No message file for product=RDBMS, facility=ORA
                            Wed Apr 23 06:17:21 2014
                            Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_24046.trc:
                            ORA-01017: Message 1017 not found; No message file for product=RDBMS, facility=ORA
                            Wed Apr 23 06:17:21 2014
                            Error 1017 received logging on to the standby
                            ------------------------------------------------------------
                            Check that the primary and standby are using a password file
                            and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
                            and that the SYS password is same in the password files.
                                  returning error ORA-16191
                            ------------------------------------------------------------
                            FAL[client, MRP0]: Error 16191 connecting to PCARD for fetching gap sequence
                            Wed Apr 23 06:17:21 2014
                            Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_24046.trc:
                            ORA-16191: Message 16191 not found; No message file for product=RDBMS, facility=ORA
                            Wed Apr 23 06:17:51 2014
                            Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_24046.trc:
                            ORA-01017: Message 1017 not found; No message file for product=RDBMS, facility=ORA
                            Wed Apr 23 06:17:51 2014
                            Error 1017 received logging on to the standby
                            ------------------------------------------------------------
                            Check that the primary and standby are using a password file
                            and remote_login_passwordfile is set to SHARED or EXCLUSIVE,
                            and that the SYS password is same in the password files.
                                  returning error ORA-16191
                            ------------------------------------------------------------
                            FAL[client, MRP0]: Error 16191 connecting to PCARD for fetching gap sequence
                            Wed Apr 23 06:17:51 2014
                            Errors in file /pcard18/admin/p_stdy/bdump/p_stdy_mrp0_24046.trc:
                            ORA-16191: Message 16191 not found; No message file for product=RDBMS, facility=ORA
                            pwcd-stage-dr:/pcard18/admin/p_stdy/bdump>
                            
                            
                            • 11. Re: Heartbeat failed to connect to standby . Error is 16058
                              yoonas

                              User348341,


                              ORA-03113: end-of-file on communication channel

                               

                              $ oerr ora 3113

                              03113, 00000, "end-of-file on communication channel"

                              // *Cause: The connection between Client and Server process was broken.

                              // *Action: There was a communication error that requires further investigation.

                              //          First, check for network problems and review the SQL*Net setup.

                              //          Also, look in the alert.log file for any errors. Finally, test to

                              //          see whether the server process is dead and whether a trace file

                              //          was generated at failure time.

                               

                               

                              Check if any network issue is happening do a continuous ping to standby from primary for sometime and see

                               

                              ORA-01017: Message 1017 not found; No message file for product=RDBMS, facility=ORA 

                              Error 1017 received logging on to the standby 

                              This error from log file shows that primary is not able to log in to standby for successful redo log shipping, Can you try recreating the password file with same password on both

                               

                               

                              Please post exact commands you run to start your standby in mount mode including environment variables(ORACLE_HOME,ORACLE_SID,..)

                               

                              What is the value for LOG_ARCHIVE_DEST_2 on primary

                               

                              Also post listener on standby

                               

                              Regards

                              Yoonas

                              • 12. Re: Heartbeat failed to connect to standby . Error is 16058
                                User348341

                                Thanks Yoonas,

                                A continous ping was sent successfully to the Standy. I have also re- created the passwordfile at both ends using the same password.

                                Also, find below the command use to put the standby database in mount mode.

                                 

                                SQL> shutdown immediate;
                                
                                
                                SQL> shutdown immediate;
                                ORA-01109: database not open
                                
                                
                                
                                
                                Database dismounted.
                                ORACLE instance shut down.
                                SQL> startup mount
                                ORACLE instance started.
                                
                                
                                Total System Global Area 1207959552 bytes
                                Fed Size                  1267164 bytes
                                Variable Size             385878564 bytes
                                Database Buffers          805306368 bytes
                                Redo Buffers               15507456 bytes
                                Database mounted.
                                SQL> alter database recover managed standby database disconnect;
                                
                                
                                Database altered.
                                

                                 

                                The value for my LOG_ARCHIVE_DEST_2 in Primary DB is as below

                                SQL> show parameter LOG_ARCHIVE_DEST_2
                                
                                
                                NAME                             TYPE        VALUE
                                ------------------------------------ ----------- ------------------------------
                                log_archive_dest_2               string      service=p_stdy LGWR ASYNC valid_for=(ONLINE_LOGFILE,PRIMARY_ROLE) db_unique_name=p_stdy
                                SQL>
                                

                                 

                                The Standby Listener is as shown below:

                                LISTENER=
                                  (DESCRIPTION_LIST =
                                    (DESCRIPTION =
                                      (ADDRESS_LIST =
                                        (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC))
                                      )
                                      (ADDRESS_LIST =
                                        (ADDRESS = (PROTOCOL = TCP)(HOST =xyz.xy.x.xyz)(PORT = 1522))
                                      )
                                    )
                                    (DESCRIPTION =
                                      (PROTOCOL_STACK =
                                        (PRESENTATION = GIOP)
                                        (SESSION = RAW)
                                      )
                                      (ADDRESS = (PROTOCOL = TCP)(HOST = xyz.xy.x.xyz)(PORT = 2481))
                                    )
                                  )
                                
                                
                                SID_LIST_LISTENER =
                                  (SID_LIST =
                                    (SID_DESC =
                                    (ENVS="EXTPROC_DLLS=ANY,LD_LIBRARY_PATH=/pcard16/pwrcard/usr/lib,MYPATH=/usr/ucb:/pcard16/pwrcard/usr/bin,APL_ENV_FILE=/pcard17/trace/env.txt")
                                      (SID_NAME = PLSExtProc)
                                      (ORACLE_HOME = /oracle/product/10.2.0)
                                      (PROGRAM = extproc)
                                    )
                                    (SID_DESC =
                                      (GLOBAL_DBNAME = p_stdy)
                                      (ORACLE_HOME = /oracle/product/10.2.0)
                                      (SID_NAME = p_stdy)
                                    )
                                  )
                                

                                 

                                I will also like to point out that the DB_NAME for both Primary and Standby is the same (PCARD)

                                 

                                Regards

                                • 13. Re: Heartbeat failed to connect to standby . Error is 16058
                                  yoonas

                                  Hi,

                                   

                                  From the standby database terminal where you started database

                                  echo ORACLE_HOME



                                  Regards

                                  Yoonas

                                  • 14. Re: Heartbeat failed to connect to standby . Error is 16058
                                    User348341

                                    Thanks Yoonas for your quick response.

                                    Please, find below for your request

                                     

                                    computer_name:/oracle/product/10.2.0/network/admin>echo $ORACLE_HOME
                                    /oracle/product/10.2.0
                                    
                                    1 2 Previous Next