7 Replies Latest reply: Aug 1, 2012 11:11 AM by SumathyP RSS

    Errors after successful automatic failover in 11.2.0.2.

    SumathyP
      After successful failover get these errors in show configuration.

      Problem Description: ORA-16829: fast-start failover configuration is lagging
      Cause: The fast-start failover target standby database was not within the lag limit
      specified by the FastStartFailoverLagLimit configuration property. As a result, a
      fast-start failover could not happen in the event of a primary database failure.
      Action: Ensure that the fast-start failover target standby database is running and
      applying redo data and that the primary database is successfully trasmitting redo
      data. If this condition persists consider raising the value of the
      FastStartFailoverLagLimit configuration property.


      Alert log has the following errors

      select status,gap_status from v$archive_dest_status where dest_id=2;

      STATUS GAP_STATUS
      --------- ------------------------
      VALID NO GAP

      SQL> select max(sequence#) from v$log_history;


      return s the same result on primary/secondary.

      However my alert log still has the same errors :

      *************************************************************
      WARNING: A file of type ARCHIVED LOG may exist in
      db_recovery_file_dest that is not known to the database.
      Use the RMAN command CATALOG RECOVERY AREA to re-catalog
      any such files. If files cannot be cataloged, then manually
      delete them using OS command. This is most likely the
      result of a crash during file creation.
      *************************************************************
      Wed Nov 30 21:47:49 2011
      krsk_srl_archive_int: Enabling archival of deferred physical standby SRLs
      Errors in file /data1/oracle/admin/calabp01/diag/rdbms/calabp01/CALABP01/trace/CALABP01_arc4_3106.trc:
      ORA-19816: WARNING: Files may exist in db_recovery_file_dest that are not known to database.
      ORA-27038: created file already exists

      Any help is appreciated.
        • 1. Re: Errors after successful automatic failover in 11.2.0.2.
          mseberg
          If you are getting the error then your Standby is behind the Primary more than your defined limit.

          What you need to understand is why your Standby is that far behind the Primary.

          Use v$dataguard_stats on the Standby database.

          Ex.
          SELECT 
            NAME,
            VALUE 
          FROM 
            V$DATAGUARD_STATS 
          WHERE 
            NAME LIKE 'transport%';
          I am unable to recreate your RMAN Error on my test system.
          CATALOG ARCHIVELOG '/u01/app/oracle/flash_recovery_area/PRIMARY/archivelog/2011_12_05/o1_mf_1_3034_7fv4wdlt_.arc';
          Best Regards

          mseberg
          • 2. Re: Errors after successful automatic failover in 11.2.0.2.
            SumathyP
            Thanks a lot for your prompt reply.

            My flashback archivelog has some files but i am not able to catalog them - can I simply get rid of them?

            /data1/oracle/oradata/flashback/CALABP01/archivelog/2011_12_05
            [oracle@mil-ora2 2011_12_05]$ ls -lrt
            total 72992
            -rw-r----- 1 oracle dba 2074624 Dec 5 22:12 o1_mf_1_17_7ftjhbhl_.arc
            -rw-r----- 1 oracle dba 771584 Dec 5 22:38 o1_mf_1_18_7ftkydtl_.arc
            -rw-r----- 1 oracle dba 1024 Dec 5 22:47 o1_mf_1_22_7ftlj7gx_.arc
            -rw-r----- 1 oracle dba 286208 Dec 5 22:56 o1_mf_1_23_7ftm1qtt_.arc
            -rw-r----- 1 oracle dba 4096 Dec 5 22:57 o1_mf_1_24_7ftm1y4y_.arc
            -rw-r----- 1 oracle dba 62464 Dec 5 22:59 o1_mf_1_25_7ftm62yb_.arc
            -rw-r----- 1 oracle dba 2048 Dec 5 22:59 o1_mf_1_26_7ftm6k43_.arc
            -rw-r----- 1 oracle dba 2115584 Dec 5 22:59 o1_mf_1_27_7ftm70kz_.arc
            -rw-r----- 1 oracle dba 5241856 Dec 5 22:59 o1_mf_1_28_7ftm78wl_.arc
            -rw-r----- 1 oracle dba 5241856 Dec 5 22:59 o1_mf_1_29_7ftm7hfd_.arc
            -rw-r----- 1 oracle dba 5510656 Dec 5 23:04 o1_mf_1_30_7ftmhqo5_.arc
            -rw-r----- 1 oracle dba 5268480 Dec 5 23:04 o1_mf_1_31_7ftmhtnj_.arc
            -rw-r----- 1 oracle dba 5271552 Dec 5 23:04 o1_mf_1_32_7ftmhxln_.arc
            -rw-r----- 1 oracle dba 5279232 Dec 5 23:04 o1_mf_1_33_7ftmj0nj_.arc
            -rw-r----- 1 oracle dba 5345792 Dec 5 23:04 o1_mf_1_34_7ftmj3nf_.arc
            -rw-r----- 1 oracle dba 5441536 Dec 5 23:04 o1_mf_1_35_7ftmj6oq_.arc
            -rw-r----- 1 oracle dba 5348352 Dec 5 23:04 o1_mf_1_36_7ftmj9m2_.arc
            -rw-r----- 1 oracle dba 5324288 Dec 5 23:04 o1_mf_1_37_7ftmjdmc_.arc
            -rw-r----- 1 oracle dba 5304320 Dec 5 23:04 o1_mf_1_38_7ftmjhp6_.arc
            -rw-r----- 1 oracle dba 5290496 Dec 5 23:04 o1_mf_1_39_7ftmjlm7_.arc
            -rw-r----- 1 oracle dba 5342720 Dec 5 23:04 o1_mf_1_40_7ftmjomw_.arc
            [oracle@mil-ora2 2011_12_05]$


            RMAN> CATALOG ARCHIVELOG '/data1/oracle/oradata/flashback/CALABP01/archivelog/20 11_12_05/o1_mf_1_40_7ftmjomw_.arc';

            RMAN-00571: ===========================================================
            RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
            RMAN-00571: ===========================================================
            RMAN-03009: failure of catalog command on default channel at 12/06/2011 18:50:42
            ORA-19625: error identifying file /data1/oracle/oradata/flashback/CALABP01/archivelog/20 11_12_05/o1_mf_1_40_7ftmjomw_.arc
            ORA-27037: unable to obtain file status
            Linux-x86_64 Error: 2: No such file or directory
            Additional information: 3

            On secondary

            SQL> SELECT
            NAME,
            VALUE
            FROM
            V$DATAGUARD_STATS
            WHERE
            NAME LIKE 'transport%';

            2 3 4 5 6 7
            NAME
            --------------------------------
            VALUE
            ----------------------------------------------------------------
            transport lag
            +00 01:09:03
            • 3. Re: Errors after successful automatic failover in 11.2.0.2.
              mseberg
              Hello again;

              Yes you should be able to. I would probably test on one file and then do an Archive Crosscheck to see if it barks

              crosscheck archivelog all;

              If it does not then delete the rest.

              Generally a crosscheck and a delete in RMAN will fix the issue, but only use if you have to.

              crosscheck archivelog all;
              delete noprompt expired archivelog all;

              Also before running this make sure you RMAN policy is set correctly ( To Applied on standby )

              Most likely RMAN does not know about these and Archive Crosscheck won't report them.

              Given how new they are I might wait a day or two just to be safe.

              Best Regards

              mseberg
              • 4. Re: Errors after successful automatic failover in 11.2.0.2.
                SumathyP
                It worked once I deleted the mysterious log files and crosschecked/deleted in rman and I could see the lag reducing in standby - then suddenly those logs again appeared in flashback and I have the same issue.

                If you specify log_archive_dest_1 then logs should not be archive to db recovery area right?
                • 5. Re: Errors after successful automatic failover in 11.2.0.2.
                  mseberg
                  If I understand correctly : ( I hope you are not saying the same logs with the same sequence numbers are coming back )

                  log_archive_dest_1

                  I would expect new logs to get created there ( based on my setup ) and "log_archive_dest_2" be set to the Standby location ( when in the Primary role).

                  For example on my Primary test :

                  log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=PRIMARY'
                  log_archive_dest_2='SERVICE=STANDBY LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=STANDBY' 
                  LOG_ARCHIVE_DEST_STATE_1=ENABLE
                  LOG_ARCHIVE_DEST_STATE_2=ENABLE
                  And on my Standby test :
                  log_archive_dest_1='LOCATION=USE_DB_RECOVERY_FILE_DEST VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=STANDBY'
                  log_archive_dest_2='SERVICE=PRIMARY LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=PRIMARY'
                  LOG_ARCHIVE_DEST_STATE_1=ENABLE
                  LOG_ARCHIVE_DEST_STATE_2=DEFER
                  The way I have mine setup in I do RMAN on the Primary, RMAN deletes my archive after a few days ( I'm Paranoid ). On the Standby I have RMAN remove
                  archive without doing a backup again keeping a few days ( again I'm Paranoid ). I have the RMAN policy set according to Oracle documents "Applied on Standby" being the main one I remember. If I did a failover I would have to review my RMAN setting as i know they are different for Primary and Standby.

                  Anyway, what unexpected behavior is occurring compared to your RMAN and Parameter settings?

                  I would create a NEW pfile form Spfile and check everything again. If you are using Broker it might add log_archive_dest_3. ( I assume you are. )

                  Best Regards

                  mseberg
                  • 6. Re: Errors after successful automatic failover in 11.2.0.2.
                    SumathyP
                    Thanks

                    Edited by: 900956 on Dec 6, 2011 1:05 PM