This discussion is archived
7 Replies Latest reply: Aug 1, 2012 9:11 AM by User375875 RSS

Errors after successful automatic failover in 11.2.0.2.

User375875 Newbie
Currently Being Moderated
After successful failover get these errors in show configuration.

Problem Description: ORA-16829: fast-start failover configuration is lagging
Cause: The fast-start failover target standby database was not within the lag limit
specified by the FastStartFailoverLagLimit configuration property. As a result, a
fast-start failover could not happen in the event of a primary database failure.
Action: Ensure that the fast-start failover target standby database is running and
applying redo data and that the primary database is successfully trasmitting redo
data. If this condition persists consider raising the value of the
FastStartFailoverLagLimit configuration property.


Alert log has the following errors

select status,gap_status from v$archive_dest_status where dest_id=2;

STATUS GAP_STATUS
--------- ------------------------
VALID NO GAP

SQL> select max(sequence#) from v$log_history;


return s the same result on primary/secondary.

However my alert log still has the same errors :

*************************************************************
WARNING: A file of type ARCHIVED LOG may exist in
db_recovery_file_dest that is not known to the database.
Use the RMAN command CATALOG RECOVERY AREA to re-catalog
any such files. If files cannot be cataloged, then manually
delete them using OS command. This is most likely the
result of a crash during file creation.
*************************************************************
Wed Nov 30 21:47:49 2011
krsk_srl_archive_int: Enabling archival of deferred physical standby SRLs
Errors in file /data1/oracle/admin/calabp01/diag/rdbms/calabp01/CALABP01/trace/CALABP01_arc4_3106.trc:
ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are not known to database.
ORA-27038: created file already exists

Any help is appreciated.
  • 1. Re: Errors after successful automatic failover in 11.2.0.2.
    mseberg Guru
    Currently Being Moderated
    If you are getting the error then your Standby is behind the Primary more than your defined limit.

    What you need to understand is why your Standby is that far behind the Primary.

    Use v$dataguard_stats on the Standby database.

    Ex.
    SELECT 
      NAME,
      VALUE 
    FROM 
      V$DATAGUARD_STATS 
    WHERE 
      NAME LIKE 'transport%';
    I am unable to recreate your RMAN Error on my test system.
    CATALOG ARCHIVELOG '/u01/app/oracle/flash_recovery_area/PRIMARY/archivelog/2011_12_05/o1_mf_1_3034_7fv4wdlt_.arc';
    Best Regards

    mseberg
  • 2. Re: Errors after successful automatic failover in 11.2.0.2.
    User375875 Newbie
    Currently Being Moderated
    Thanks a lot for your prompt reply.

    My flashback archivelog has some files but i am not able to catalog them - can I simply get rid of them?

    /data1/oracle/oradata/flashback/CALABP01/archivelog/2011_12_05
    [oracle@mil-ora2 2011_12_05]$ ls -lrt
    total 72992
    -rw-r----- 1 oracle dba 2074624 Dec 5 22:12 o1_mf_1_17_7ftjhbhl_.arc
    -rw-r----- 1 oracle dba 771584 Dec 5 22:38 o1_mf_1_18_7ftkydtl_.arc
    -rw-r----- 1 oracle dba 1024 Dec 5 22:47 o1_mf_1_22_7ftlj7gx_.arc
    -rw-r----- 1 oracle dba 286208 Dec 5 22:56 o1_mf_1_23_7ftm1qtt_.arc
    -rw-r----- 1 oracle dba 4096 Dec 5 22:57 o1_mf_1_24_7ftm1y4y_.arc
    -rw-r----- 1 oracle dba 62464 Dec 5 22:59 o1_mf_1_25_7ftm62yb_.arc
    -rw-r----- 1 oracle dba 2048 Dec 5 22:59 o1_mf_1_26_7ftm6k43_.arc
    -rw-r----- 1 oracle dba 2115584 Dec 5 22:59 o1_mf_1_27_7ftm70kz_.arc
    -rw-r----- 1 oracle dba 5241856 Dec 5 22:59 o1_mf_1_28_7ftm78wl_.arc
    -rw-r----- 1 oracle dba 5241856 Dec 5 22:59 o1_mf_1_29_7ftm7hfd_.arc
    -rw-r----- 1 oracle dba 5510656 Dec 5 23:04 o1_mf_1_30_7ftmhqo5_.arc
    -rw-r----- 1 oracle dba 5268480 Dec 5 23:04 o1_mf_1_31_7ftmhtnj_.arc
    -rw-r----- 1 oracle dba 5271552 Dec 5 23:04 o1_mf_1_32_7ftmhxln_.arc
    -rw-r----- 1 oracle dba 5279232 Dec 5 23:04 o1_mf_1_33_7ftmj0nj_.arc
    -rw-r----- 1 oracle dba 5345792 Dec 5 23:04 o1_mf_1_34_7ftmj3nf_.arc
    -rw-r----- 1 oracle dba 5441536 Dec 5 23:04 o1_mf_1_35_7ftmj6oq_.arc
    -rw-r----- 1 oracle dba 5348352 Dec 5 23:04 o1_mf_1_36_7ftmj9m2_.arc
    -rw-r----- 1 oracle dba 5324288 Dec 5 23:04 o1_mf_1_37_7ftmjdmc_.arc
    -rw-r----- 1 oracle dba 5304320 Dec 5 23:04 o1_mf_1_38_7ftmjhp6_.arc
    -rw-r----- 1 oracle dba 5290496 Dec 5 23:04 o1_mf_1_39_7ftmjlm7_.arc
    -rw-r----- 1 oracle dba 5342720 Dec 5 23:04 o1_mf_1_40_7ftmjomw_.arc
    [oracle@mil-ora2 2011_12_05]$


    RMAN> CATALOG ARCHIVELOG '/data1/oracle/oradata/flashback/CALABP01/archivelog/20 11_12_05/o1_mf_1_40_7ftmjomw_.arc';

    RMAN-00571: ===========================================================
    RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
    RMAN-00571: ===========================================================
    RMAN-03009: failure of catalog command on default channel at 12/06/2011 18:50:42
    ORA-19625: error identifying file /data1/oracle/oradata/flashback/CALABP01/archivelog/20 11_12_05/o1_mf_1_40_7ftmjomw_.arc
    ORA-27037: unable to obtain file status
    Linux-x86_64 Error: 2: No such file or directory
    Additional information: 3

    On secondary

    SQL> SELECT
    NAME,
    VALUE
    FROM
    V$DATAGUARD_STATS
    WHERE
    NAME LIKE 'transport%';

    2 3 4 5 6 7
    NAME
    --------------------------------
    VALUE
    ----------------------------------------------------------------
    transport lag
    +00 01:09:03
  • 3. Re: Errors after successful automatic failover in 11.2.0.2.
    mseberg Guru
    Currently Being Moderated
    Hello again;

    Yes you should be able to. I would probably test on one file and then do an Archive Crosscheck to see if it barks

    crosscheck archivelog all;

    If it does not then delete the rest.

    Generally a crosscheck and a delete in RMAN will fix the issue, but only use if you have to.

    crosscheck archivelog all;
    delete noprompt expired archivelog all;

    Also before running this make sure you RMAN policy is set correctly ( To Applied on standby )

    Most likely RMAN does not know about these and Archive Crosscheck won't report them.

    Given how new they are I might wait a day or two just to be safe.

    Best Regards

    mseberg
  • 4. Re: Errors after successful automatic failover in 11.2.0.2.
    User375875 Newbie
    Currently Being Moderated
    It worked once I deleted the mysterious log files and crosschecked/deleted in rman and I could see the lag reducing in standby - then suddenly those logs again appeared in flashback and I have the same issue.

    If you specify log_archive_dest_1 then logs should not be archive to db recovery area right?
  • 5. Re: Errors after successful automatic failover in 11.2.0.2.
    mseberg Guru
    Currently Being Moderated
    If I understand correctly : ( I hope you are not saying the same logs with the same sequence numbers are coming back )

    log_archive_dest_1

    I would expect new logs to get created there ( based on my setup ) and "log_archive_dest_2" be set to the Standby location ( when in the Primary role).

    For example on my Primary test :

    log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=PRIMARY'
    log_archive_dest_2='SERVICE=STANDBY LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=STANDBY' 
    LOG_ARCHIVE_DEST_STATE_1=ENABLE
    LOG_ARCHIVE_DEST_STATE_2=ENABLE
    And on my Standby test :
    log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=STANDBY'
    log_archive_dest_2='SERVICE=PRIMARY LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PRIMARY'
    LOG_ARCHIVE_DEST_STATE_1=ENABLE
    LOG_ARCHIVE_DEST_STATE_2=DEFER
    The way I have mine setup in I do RMAN on the Primary, RMAN deletes my archive after a few days ( I'm Paranoid ). On the Standby I have RMAN remove
    archive without doing a backup again keeping a few days ( again I'm Paranoid ). I have the RMAN policy set according to Oracle documents "Applied on Standby" being the main one I remember. If I did a failover I would have to review my RMAN setting as i know they are different for Primary and Standby.

    Anyway, what unexpected behavior is occurring compared to your RMAN and Parameter settings?

    I would create a NEW pfile form Spfile and check everything again. If you are using Broker it might add log_archive_dest_3. ( I assume you are. )

    Best Regards

    mseberg
  • 6. Re: Errors after successful automatic failover in 11.2.0.2.
    User375875 Newbie
    Currently Being Moderated
    Thanks

    Edited by: 900956 on Dec 6, 2011 1:05 PM
  • 7. Re: Errors after successful automatic failover in 11.2.0.2.
    User375875 Newbie
    Currently Being Moderated
    Thanks losing

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points