1 2 Previous Next 19 Replies Latest reply: Dec 6, 2012 3:48 AM by adnanKaysar RSS

    PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.

    adnanKaysar
      Hi all,
      Good day.

      I have recently configured oracle data guard on centos 6.2 and oracle 11g.

      The password file created in primary and copied to the standby location $ORACLE_HOME/dbs but primary sys user cant connect to standby.

      Thanks,
      Adnan
        • 1. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
          Shivananda Rao
          Hello,
          The password file created in primary and copied to the standby location $ORACLE_HOME/dbs but primary sys user cant connect to standby.
          After copying the password file from primary server to standby server, did you rename it to "orapw<Standby-SID>" on the standby server ?

          Make sure that the parameter "remote_login_passwordfile=EXCLUSIVE" on both primary and standby database SPFILE.

          Also, from the primary database, please post the outcome of the below query:
          SQL>select severity,error_code,message,timestamp from v$dataguard_status where dest_id=2;
          • 2. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
            adnanKaysar
            Hi the output is below:
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:09:37 PM
            Error     16191     Error 16191 for archive log file 3 to 'rndstdby'     12/6/2012 12:09:37 PM
            Error     16191     FAL[server, ARC0]: Error 16191 creating remote archivelog file 'rndstdby'     12/6/2012 12:09:37 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:15:33 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:21:33 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:27:33 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:33:33 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:39:33 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:45:33 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:50:34 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:56:34 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:02:34 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:08:34 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:14:34 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:20:34 PM
            Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:26:35 PM
            remote_login_pawword_file set to EXCLUSIVE in both dbs
            • 3. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
              CKPT
              adnanKaysar wrote:
              Hi the output is below:
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:09:37 PM
              Error     16191     Error 16191 for archive log file 3 to 'rndstdby'     12/6/2012 12:09:37 PM
              Error     16191     FAL[server, ARC0]: Error 16191 creating remote archivelog file 'rndstdby'     12/6/2012 12:09:37 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:15:33 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:21:33 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:27:33 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:33:33 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:39:33 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:45:33 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:50:34 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:56:34 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:02:34 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:08:34 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:14:34 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:20:34 PM
              Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:26:35 PM
              remote_login_pawword_file set to EXCLUSIVE in both dbs
              16191, 0000, "Primary log shipping client not logged on standby"
              // *Cause:  An attempt to ship redo to standby without logging on
              // to standby or with invalid user credentials.
              // *Action: Check that primary and standby are using password files and that
              //          both primary and standby have the same SYS password.
              //          Restart primary and/or standby after ensuring that
              //          password file is accessible and REMOTE_LOGIN_PASSWORDFILE
              //          initialization parameter is set to SHARED or EXCLUSIVE.
              It clears that issue is with password file, even if you set REMOTE_LOGIN_PASSWORDFILE to Exclusive.

              Are you sure that you able to connect from primary to standby and vice versa using oracle Net service?
              Ex from Primary: $sqlplus sys/*****@standby as sysdba

              Can you share your password file full name? Have you mentioned any extensions?

              and use query from primary and standby
              SQL> select * from v$pwfile_users;
              • 4. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
                Osama_Mustafa
                This is error is totally clear , check MOS Notes :
                Troubleshooting - Heartbeat failed to connect to standby [ID 1432367.1]
                • 5. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
                  Shivananda Rao
                  Hello,

                  What about the password file format that I asked ? After copying the password file to the standby server, did you rename it ?


                  Regards,
                  Shivananda
                  • 6. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
                    adnanKaysar
                    From standby to primary sys can connect.

                    From primary db to standby db:
                    [oracle@rndmain dbs]$ sqlplus sys/*****@rndstdby as sysdba
                    
                    SQL*Plus: Release 11.2.0.1.0 Production on Thu Dec 6 13:46:23 2012
                    
                    Copyright (c) 1982, 2009, Oracle.  All rights reserved.
                    
                    ERROR:
                    ORA-01031: insufficient privileges
                    
                    
                    Both show the same result below:
                    
                    SQL> select * from v$pwfile_users;
                    
                    USERNAME                       SYSDB SYSOP SYSAS
                    ------------------------------ ----- ----- -----
                    SYS                            TRUE  TRUE  FALSE
                    • 7. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
                      adnanKaysar
                      hi Shivananda Rao,

                      I have renamed the file.

                      thanks
                      • 8. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
                        CKPT
                        [oracle@rndmain dbs]$ sqlplus sys/*****@rndstdby as sysdba
                        
                        SQL*Plus: Release 11.2.0.1.0 Production on Thu Dec 6 13:46:23 2012
                        
                        Copyright (c) 1982, 2009, Oracle.  All rights reserved.
                        
                        ERROR:
                        ORA-01031: insufficient privileges
                        This is enough to prove that your password file is not working. What is the name and SID of the standby?
                        it should be as below.

                        cd $ORACLE_HOME/dbs
                        $orapwd file=orapw$ORACLE_SID password=****** force=y entries=10
                        Note:= Hope you already have ORACLE_SID in place and no extension with the file.

                        Now retry the operations to connect.
                        • 9. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
                          adnanKaysar
                          Hi CKPT,

                          Password file is ok created as such way you mentioned but now its another error there in alert log file , ORA-01034: ORACLE not available but standby is in MOUNT state.
                          Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
                          ORA-16191: Primary log shipping client not logged on standby
                          PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.
                          Thu Dec 06 13:44:36 2012
                          Error 1034 received logging on to the standby
                          Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
                          ORA-01034: ORACLE not available
                          PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 1034.
                          Thu Dec 06 13:50:36 2012
                          Error 1034 received logging on to the standby
                          Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
                          ORA-01034: ORACLE not available
                          PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 1034.
                          Thu Dec 06 13:50:44 2012
                          Using STANDBY_ARCHIVE_DEST parameter default value as /usr/u02/archivelogs/rndmain/
                          ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=MEMORY SID='*';
                          Error 1034 received logging on to the standby
                          Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
                          ORA-01034: ORACLE not available
                          PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 1034.
                          Thu Dec 06 13:50:46 2012
                          Thread 1 advanced to log sequence 170 (LGWR switch)
                            Current log# 2 seq# 170 mem# 0: /u01/app/oracle/oradata/rndmain/redo02a.log
                            Current log# 2 seq# 170 mem# 1: /usr/u02/oradata/rndmain/redo02b.log
                          Thu Dec 06 13:50:47 2012
                          Archived Log entry 45 added for thread 1 sequence 169 ID 0x72c2dbea dest 1:
                          Thu Dec 06 13:56:36 2012
                          Error 1034 received logging on to the standby
                          Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
                          ORA-01034: ORACLE not available
                          PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 1034.
                          • 10. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
                            CKPT
                            Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
                            ORA-01034: ORACLE not available
                            PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 1034.
                            So instance is crashed, What is the status of standby now? what is latest update in standby alert log file. Do you think allocated enough SGA for the instance?
                            Review alert log file and fix it. If it is down bring standby database in MOUNT status and start MRP.

                            If your standby is not available, I have a doubt. how you got below output?
                            SQL> select * from v$pwfile_users;
                            
                            USERNAME                       SYSDB SYSOP SYSAS
                            ------------------------------ ----- ----- -----
                            SYS                            TRUE  TRUE  FALSE
                            Probably you queried it in Primary? :(
                            • 11. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
                              Shivananda Rao
                              Bring down the standby database and start it up either in mount stage or open mode.
                              Once done, start the MRP process on the standby.


                              Regards,
                              Shivananda
                              • 12. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
                                adnanKaysar
                                Hi,

                                I ran the query select * from v$pwfile_users; in both instance;

                                Standby is up and running in MOUNT mode and MRP started but in primary alert log its showing its down.

                                SGA is 2GB here in standby and 4G in primary.

                                Thanks
                                • 13. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
                                  adnanKaysar
                                  Hi,

                                  ALready tried it after noticed the Oracle not available but no hope .

                                  satandby log is below:

                                  {copy}
                                  Instance shutdown complete
                                  Thu Dec 06 13:46:06 2012
                                  Starting ORACLE instance (normal)
                                  LICENSE_MAX_SESSION = 0
                                  LICENSE_SESSIONS_WARNING = 0
                                  Picked latch-free SCN scheme 3
                                  Autotune of undo retention is turned on.
                                  IMODE=BR
                                  ILAT =68
                                  LICENSE_MAX_USERS = 0
                                  SYS auditing is disabled
                                  Starting up:
                                  Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
                                  With the Partitioning, Oracle Label Security, OLAP, Data Mining,
                                  Oracle Database Vault and Real Application Testing options.
                                  Using parameter settings in server-side spfile /u01/app/oracle/product/11.2.0/dbhome_1/dbs/spfilerndstdby.ora
                                  System parameters with non-default values:
                                  processes = 400
                                  sga_max_size = 2000M
                                  sga_target = 2000M
                                  control_files = "/data/rndstdby/rndmain/control01.ctl"
                                  control_files = "/usr/u02/oradata/rndstdby/control02.ctl"
                                  db_file_name_convert = "/u01/app/oracle/oradata/"
                                  db_file_name_convert = "/data/rndstdby/"
                                  log_file_name_convert = "/usr/u02/archivelogs/rndmain"
                                  log_file_name_convert = "/usr/u02/archivelogs/rndstdby"
                                  db_block_size = 8192
                                  db_writer_processes = 2
                                  compatible = "11.2.0"
                                  log_archive_dest_1 = "LOCATION=/usr/u02/archivelogs/rndstdby/ VALID_FOR=(ALL_LOGFILES,ALL_ROLES)"
                                  log_archive_dest_2 = "SERVICE=rndmain ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=rndmain"
                                  log_archive_dest_state_1 = "enable"
                                  fal_server = "rndmain"
                                  log_archive_config = "DG_CONFIG=(rndmain,rndstdby)"
                                  log_archive_format = "rnddb_%t_%s_%r.arc"
                                  log_archive_max_processes= 5
                                  standby_file_management = "auto"
                                  fast_start_mttr_target = 3600
                                  undo_tablespace = "UNDOTBS1"
                                  sec_case_sensitive_logon = FALSE
                                  remote_login_passwordfile= "EXCLUSIVE"
                                  db_domain = ""
                                  dispatchers = "(PROTOCOL=TCP) (SERVICE=RNDMAINXDB)"
                                  shared_servers = 2
                                  max_shared_servers = 3
                                  audit_file_dest = "/u01/app/oracle/admin/rndstdby/adump"
                                  audit_trail = "DB"
                                  db_name = "rndmain"
                                  db_unique_name = "rndstdby"
                                  open_cursors = 300
                                  diagnostic_dest = "/u01/app/oracle/diag/rndmain"
                                  Thu Dec 06 13:46:06 2012
                                  PMON started with pid=2, OS id=3641
                                  Thu Dec 06 13:46:06 2012
                                  VKTM started with pid=3, OS id=3643
                                  VKTM running at (100ms) precision
                                  Thu Dec 06 13:46:06 2012
                                  GEN0 started with pid=4, OS id=3647
                                  Thu Dec 06 13:46:06 2012
                                  DIAG started with pid=5, OS id=3649
                                  Thu Dec 06 13:46:06 2012
                                  DBRM started with pid=6, OS id=3651
                                  Thu Dec 06 13:46:06 2012
                                  PSP0 started with pid=7, OS id=3653
                                  Thu Dec 06 13:46:06 2012
                                  DIA0 started with pid=8, OS id=3655
                                  Thu Dec 06 13:46:06 2012
                                  MMAN started with pid=9, OS id=3657
                                  Thu Dec 06 13:46:06 2012
                                  DBW0 started with pid=10, OS id=3659
                                  Thu Dec 06 13:46:06 2012
                                  DBW1 started with pid=11, OS id=3661
                                  Thu Dec 06 13:46:06 2012
                                  LGWR started with pid=12, OS id=3663
                                  Thu Dec 06 13:46:06 2012
                                  CKPT started with pid=13, OS id=3665
                                  Thu Dec 06 13:46:06 2012
                                  SMON started with pid=14, OS id=3667
                                  Thu Dec 06 13:46:06 2012
                                  RECO started with pid=15, OS id=3669
                                  Thu Dec 06 13:46:06 2012
                                  MMON started with pid=16, OS id=3671
                                  starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
                                  Thu Dec 06 13:46:06 2012
                                  MMNL started with pid=17, OS id=3673
                                  starting up 2 shared server(s) ...
                                  ORACLE_BASE from environment = /u01/app/oracle
                                  Thu Dec 06 13:46:06 2012
                                  ALTER DATABASE MOUNT
                                  ARCH: STARTING ARCH PROCESSES
                                  Thu Dec 06 13:46:10 2012
                                  ARC0 started with pid=22, OS id=3686
                                  ARC0: Archival started
                                  ARCH: STARTING ARCH PROCESSES COMPLETE
                                  ARC0: STARTING ARCH PROCESSES
                                  Thu Dec 06 13:46:11 2012
                                  ARC1 started with pid=23, OS id=3688
                                  Thu Dec 06 13:46:11 2012
                                  ARC2 started with pid=24, OS id=3690
                                  Thu Dec 06 13:46:11 2012
                                  ARC3 started with pid=25, OS id=3692
                                  ARC1: Archival started
                                  Thu Dec 06 13:46:11 2012
                                  ARC2: Archival startedARC4 started with pid=26, OS id=3694

                                  ARC3: Archival started
                                  ARC1: Becoming the 'no FAL' ARCH
                                  ARC1: Becoming the 'no SRL' ARCH
                                  ARC2: Becoming the heartbeat ARCH
                                  Successful mount of redo thread 1, with mount id 1926560926
                                  Physical Standby Database mounted.
                                  Lost write protection disabled
                                  Completed: ALTER DATABASE MOUNT
                                  ARC4: Archival started
                                  ARC0: STARTING ARCH PROCESSES COMPLETE
                                  Thu Dec 06 13:52:42 2012
                                  alter database recover managed standby database disconnect from session
                                  Attempt to start background Managed Standby Recovery process (rndstdby)
                                  Thu Dec 06 13:52:42 2012
                                  MRP0 started with pid=27, OS id=3724
                                  MRP0: Background Managed Standby Recovery process started (rndstdby)
                                  started logmerger process
                                  Thu Dec 06 13:52:48 2012
                                  Managed Standby Recovery not using Real Time Apply
                                  Parallel Media Recovery started with 2 slaves
                                  Waiting for all non-current ORLs to be archived...
                                  All non-current ORLs have been archived.
                                  Media Recovery Waiting for thread 1 sequence 164
                                  Completed: alter database recover managed standby database disconnect from session
                                  {copy}
                                  • 14. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
                                    CKPT
                                    adnanKaysar wrote:
                                    Hi,

                                    I ran the query select * from v$pwfile_users; in both instance;

                                    Standby is up and running in MOUNT mode and MRP started but in primary alert log its showing its down.

                                    SGA is 2GB here in standby and 4G in primary.

                                    Thanks
                                    I need two outputs from you. Any recent changes at server level, From when it was not functioning?

                                    From Primary:-
                                    $tnsping rndstdby

                                    From standby:-
                                    $cat $ORACLE_HOME/network/admin/listener.ora
                                    $lsnrctl status

                                    Note:-If you have license of ADG then only start MRP from Open.
                                    1 2 Previous Next