This discussion is archived
1 2 Previous Next 19 Replies Latest reply: Dec 6, 2012 1:48 AM by adnanKaysar RSS

PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.

adnanKaysar Explorer
Currently Being Moderated
Hi all,
Good day.

I have recently configured oracle data guard on centos 6.2 and oracle 11g.

The password file created in primary and copied to the standby location $ORACLE_HOME/dbs but primary sys user cant connect to standby.

Thanks,
Adnan
  • 1. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    Shivananda Rao Guru
    Currently Being Moderated
    Hello,
    The password file created in primary and copied to the standby location $ORACLE_HOME/dbs but primary sys user cant connect to standby.
    After copying the password file from primary server to standby server, did you rename it to "orapw<Standby-SID>" on the standby server ?

    Make sure that the parameter "remote_login_passwordfile=EXCLUSIVE" on both primary and standby database SPFILE.

    Also, from the primary database, please post the outcome of the below query:
    SQL>select severity,error_code,message,timestamp from v$dataguard_status where dest_id=2;
  • 2. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    adnanKaysar Explorer
    Currently Being Moderated
    Hi the output is below:
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:09:37 PM
    Error     16191     Error 16191 for archive log file 3 to 'rndstdby'     12/6/2012 12:09:37 PM
    Error     16191     FAL[server, ARC0]: Error 16191 creating remote archivelog file 'rndstdby'     12/6/2012 12:09:37 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:15:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:21:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:27:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:33:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:39:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:45:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:50:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:56:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:02:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:08:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:14:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:20:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:26:35 PM
    remote_login_pawword_file set to EXCLUSIVE in both dbs
  • 3. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    CKPT Guru
    Currently Being Moderated
    adnanKaysar wrote:
    Hi the output is below:
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:09:37 PM
    Error     16191     Error 16191 for archive log file 3 to 'rndstdby'     12/6/2012 12:09:37 PM
    Error     16191     FAL[server, ARC0]: Error 16191 creating remote archivelog file 'rndstdby'     12/6/2012 12:09:37 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:15:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:21:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:27:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:33:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:39:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:45:33 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:50:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 12:56:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:02:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:08:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:14:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:20:34 PM
    Error     16191     PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.     12/6/2012 1:26:35 PM
    remote_login_pawword_file set to EXCLUSIVE in both dbs
    16191, 0000, "Primary log shipping client not logged on standby"
    // *Cause:  An attempt to ship redo to standby without logging on
    // to standby or with invalid user credentials.
    // *Action: Check that primary and standby are using password files and that
    //          both primary and standby have the same SYS password.
    //          Restart primary and/or standby after ensuring that
    //          password file is accessible and REMOTE_LOGIN_PASSWORDFILE
    //          initialization parameter is set to SHARED or EXCLUSIVE.
    It clears that issue is with password file, even if you set REMOTE_LOGIN_PASSWORDFILE to Exclusive.

    Are you sure that you able to connect from primary to standby and vice versa using oracle Net service?
    Ex from Primary: $sqlplus sys/*****@standby as sysdba

    Can you share your password file full name? Have you mentioned any extensions?

    and use query from primary and standby
    SQL> select * from v$pwfile_users;
  • 4. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    Osama_Mustafa Oracle ACE
    Currently Being Moderated
    This is error is totally clear , check MOS Notes :
    Troubleshooting - Heartbeat failed to connect to standby [ID 1432367.1]
  • 5. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    Shivananda Rao Guru
    Currently Being Moderated
    Hello,

    What about the password file format that I asked ? After copying the password file to the standby server, did you rename it ?


    Regards,
    Shivananda
  • 6. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    adnanKaysar Explorer
    Currently Being Moderated
    From standby to primary sys can connect.

    From primary db to standby db:
    [oracle@rndmain dbs]$ sqlplus sys/*****@rndstdby as sysdba
    
    SQL*Plus: Release 11.2.0.1.0 Production on Thu Dec 6 13:46:23 2012
    
    Copyright (c) 1982, 2009, Oracle.  All rights reserved.
    
    ERROR:
    ORA-01031: insufficient privileges
    
    
    Both show the same result below:
    
    SQL> select * from v$pwfile_users;
    
    USERNAME                       SYSDB SYSOP SYSAS
    ------------------------------ ----- ----- -----
    SYS                            TRUE  TRUE  FALSE
  • 7. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    adnanKaysar Explorer
    Currently Being Moderated
    hi Shivananda Rao,

    I have renamed the file.

    thanks
  • 8. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    CKPT Guru
    Currently Being Moderated
    [oracle@rndmain dbs]$ sqlplus sys/*****@rndstdby as sysdba
    
    SQL*Plus: Release 11.2.0.1.0 Production on Thu Dec 6 13:46:23 2012
    
    Copyright (c) 1982, 2009, Oracle.  All rights reserved.
    
    ERROR:
    ORA-01031: insufficient privileges
    This is enough to prove that your password file is not working. What is the name and SID of the standby?
    it should be as below.

    cd $ORACLE_HOME/dbs
    $orapwd file=orapw$ORACLE_SID password=****** force=y entries=10
    Note:= Hope you already have ORACLE_SID in place and no extension with the file.

    Now retry the operations to connect.
  • 9. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    adnanKaysar Explorer
    Currently Being Moderated
    Hi CKPT,

    Password file is ok created as such way you mentioned but now its another error there in alert log file , ORA-01034: ORACLE not available but standby is in MOUNT state.
    Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
    ORA-16191: Primary log shipping client not logged on standby
    PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 16191.
    Thu Dec 06 13:44:36 2012
    Error 1034 received logging on to the standby
    Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
    ORA-01034: ORACLE not available
    PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 1034.
    Thu Dec 06 13:50:36 2012
    Error 1034 received logging on to the standby
    Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
    ORA-01034: ORACLE not available
    PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 1034.
    Thu Dec 06 13:50:44 2012
    Using STANDBY_ARCHIVE_DEST parameter default value as /usr/u02/archivelogs/rndmain/
    ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=MEMORY SID='*';
    Error 1034 received logging on to the standby
    Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
    ORA-01034: ORACLE not available
    PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 1034.
    Thu Dec 06 13:50:46 2012
    Thread 1 advanced to log sequence 170 (LGWR switch)
      Current log# 2 seq# 170 mem# 0: /u01/app/oracle/oradata/rndmain/redo02a.log
      Current log# 2 seq# 170 mem# 1: /usr/u02/oradata/rndmain/redo02b.log
    Thu Dec 06 13:50:47 2012
    Archived Log entry 45 added for thread 1 sequence 169 ID 0x72c2dbea dest 1:
    Thu Dec 06 13:56:36 2012
    Error 1034 received logging on to the standby
    Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
    ORA-01034: ORACLE not available
    PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 1034.
  • 10. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    CKPT Guru
    Currently Being Moderated
    Errors in file /u01/app/oracle/diag/rndmain/diag/rdbms/rndmain/rndmain/trace/rndmain_arc2_11148.trc:
    ORA-01034: ORACLE not available
    PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'. Error is 1034.
    So instance is crashed, What is the status of standby now? what is latest update in standby alert log file. Do you think allocated enough SGA for the instance?
    Review alert log file and fix it. If it is down bring standby database in MOUNT status and start MRP.

    If your standby is not available, I have a doubt. how you got below output?
    SQL> select * from v$pwfile_users;
    
    USERNAME                       SYSDB SYSOP SYSAS
    ------------------------------ ----- ----- -----
    SYS                            TRUE  TRUE  FALSE
    Probably you queried it in Primary? :(
  • 11. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    Shivananda Rao Guru
    Currently Being Moderated
    Bring down the standby database and start it up either in mount stage or open mode.
    Once done, start the MRP process on the standby.


    Regards,
    Shivananda
  • 12. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    adnanKaysar Explorer
    Currently Being Moderated
    Hi,

    I ran the query select * from v$pwfile_users; in both instance;

    Standby is up and running in MOUNT mode and MRP started but in primary alert log its showing its down.

    SGA is 2GB here in standby and 4G in primary.

    Thanks
  • 13. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    adnanKaysar Explorer
    Currently Being Moderated
    Hi,

    ALready tried it after noticed the Oracle not available but no hope .

    satandby log is below:

    {copy}
    Instance shutdown complete
    Thu Dec 06 13:46:06 2012
    Starting ORACLE instance (normal)
    LICENSE_MAX_SESSION = 0
    LICENSE_SESSIONS_WARNING = 0
    Picked latch-free SCN scheme 3
    Autotune of undo retention is turned on.
    IMODE=BR
    ILAT =68
    LICENSE_MAX_USERS = 0
    SYS auditing is disabled
    Starting up:
    Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
    With the Partitioning, Oracle Label Security, OLAP, Data Mining,
    Oracle Database Vault and Real Application Testing options.
    Using parameter settings in server-side spfile /u01/app/oracle/product/11.2.0/dbhome_1/dbs/spfilerndstdby.ora
    System parameters with non-default values:
    processes = 400
    sga_max_size = 2000M
    sga_target = 2000M
    control_files = "/data/rndstdby/rndmain/control01.ctl"
    control_files = "/usr/u02/oradata/rndstdby/control02.ctl"
    db_file_name_convert = "/u01/app/oracle/oradata/"
    db_file_name_convert = "/data/rndstdby/"
    log_file_name_convert = "/usr/u02/archivelogs/rndmain"
    log_file_name_convert = "/usr/u02/archivelogs/rndstdby"
    db_block_size = 8192
    db_writer_processes = 2
    compatible = "11.2.0"
    log_archive_dest_1 = "LOCATION=/usr/u02/archivelogs/rndstdby/ VALID_FOR=(ALL_LOGFILES,ALL_ROLES)"
    log_archive_dest_2 = "SERVICE=rndmain ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=rndmain"
    log_archive_dest_state_1 = "enable"
    fal_server = "rndmain"
    log_archive_config = "DG_CONFIG=(rndmain,rndstdby)"
    log_archive_format = "rnddb_%t_%s_%r.arc"
    log_archive_max_processes= 5
    standby_file_management = "auto"
    fast_start_mttr_target = 3600
    undo_tablespace = "UNDOTBS1"
    sec_case_sensitive_logon = FALSE
    remote_login_passwordfile= "EXCLUSIVE"
    db_domain = ""
    dispatchers = "(PROTOCOL=TCP) (SERVICE=RNDMAINXDB)"
    shared_servers = 2
    max_shared_servers = 3
    audit_file_dest = "/u01/app/oracle/admin/rndstdby/adump"
    audit_trail = "DB"
    db_name = "rndmain"
    db_unique_name = "rndstdby"
    open_cursors = 300
    diagnostic_dest = "/u01/app/oracle/diag/rndmain"
    Thu Dec 06 13:46:06 2012
    PMON started with pid=2, OS id=3641
    Thu Dec 06 13:46:06 2012
    VKTM started with pid=3, OS id=3643
    VKTM running at (100ms) precision
    Thu Dec 06 13:46:06 2012
    GEN0 started with pid=4, OS id=3647
    Thu Dec 06 13:46:06 2012
    DIAG started with pid=5, OS id=3649
    Thu Dec 06 13:46:06 2012
    DBRM started with pid=6, OS id=3651
    Thu Dec 06 13:46:06 2012
    PSP0 started with pid=7, OS id=3653
    Thu Dec 06 13:46:06 2012
    DIA0 started with pid=8, OS id=3655
    Thu Dec 06 13:46:06 2012
    MMAN started with pid=9, OS id=3657
    Thu Dec 06 13:46:06 2012
    DBW0 started with pid=10, OS id=3659
    Thu Dec 06 13:46:06 2012
    DBW1 started with pid=11, OS id=3661
    Thu Dec 06 13:46:06 2012
    LGWR started with pid=12, OS id=3663
    Thu Dec 06 13:46:06 2012
    CKPT started with pid=13, OS id=3665
    Thu Dec 06 13:46:06 2012
    SMON started with pid=14, OS id=3667
    Thu Dec 06 13:46:06 2012
    RECO started with pid=15, OS id=3669
    Thu Dec 06 13:46:06 2012
    MMON started with pid=16, OS id=3671
    starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
    Thu Dec 06 13:46:06 2012
    MMNL started with pid=17, OS id=3673
    starting up 2 shared server(s) ...
    ORACLE_BASE from environment = /u01/app/oracle
    Thu Dec 06 13:46:06 2012
    ALTER DATABASE MOUNT
    ARCH: STARTING ARCH PROCESSES
    Thu Dec 06 13:46:10 2012
    ARC0 started with pid=22, OS id=3686
    ARC0: Archival started
    ARCH: STARTING ARCH PROCESSES COMPLETE
    ARC0: STARTING ARCH PROCESSES
    Thu Dec 06 13:46:11 2012
    ARC1 started with pid=23, OS id=3688
    Thu Dec 06 13:46:11 2012
    ARC2 started with pid=24, OS id=3690
    Thu Dec 06 13:46:11 2012
    ARC3 started with pid=25, OS id=3692
    ARC1: Archival started
    Thu Dec 06 13:46:11 2012
    ARC2: Archival startedARC4 started with pid=26, OS id=3694

    ARC3: Archival started
    ARC1: Becoming the 'no FAL' ARCH
    ARC1: Becoming the 'no SRL' ARCH
    ARC2: Becoming the heartbeat ARCH
    Successful mount of redo thread 1, with mount id 1926560926
    Physical Standby Database mounted.
    Lost write protection disabled
    Completed: ALTER DATABASE MOUNT
    ARC4: Archival started
    ARC0: STARTING ARCH PROCESSES COMPLETE
    Thu Dec 06 13:52:42 2012
    alter database recover managed standby database disconnect from session
    Attempt to start background Managed Standby Recovery process (rndstdby)
    Thu Dec 06 13:52:42 2012
    MRP0 started with pid=27, OS id=3724
    MRP0: Background Managed Standby Recovery process started (rndstdby)
    started logmerger process
    Thu Dec 06 13:52:48 2012
    Managed Standby Recovery not using Real Time Apply
    Parallel Media Recovery started with 2 slaves
    Waiting for all non-current ORLs to be archived...
    All non-current ORLs have been archived.
    Media Recovery Waiting for thread 1 sequence 164
    Completed: alter database recover managed standby database disconnect from session
    {copy}
  • 14. Re: PING[ARC2]: Heartbeat failed to connect to standby 'rndstdby'.
    CKPT Guru
    Currently Being Moderated
    adnanKaysar wrote:
    Hi,

    I ran the query select * from v$pwfile_users; in both instance;

    Standby is up and running in MOUNT mode and MRP started but in primary alert log its showing its down.

    SGA is 2GB here in standby and 4G in primary.

    Thanks
    I need two outputs from you. Any recent changes at server level, From when it was not functioning?

    From Primary:-
    $tnsping rndstdby

    From standby:-
    $cat $ORACLE_HOME/network/admin/listener.ora
    $lsnrctl status

    Note:-If you have license of ADG then only start MRP from Open.
1 2 Previous Next

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points