4 Replies Latest reply: Feb 16, 2014 11:47 PM by Tom321 RSS

Switch Over Failing through DGMGRL

990991 Newbie
Currently Being Moderated

I configured DGMGRL successfully and able to add standby database to DGMGRL. Now when i issue the command to swtichover to  <standbyname> its throwing the below error. Please share your suggestions  to solve this. Those who wants to suggest checking alert log adn dgmgrl log.. please find the complete log sequences below.

 

DGMGRL> switchover to db2

Performing switchover NOW, please wait...


Performing switchover NOW, please wait…

Error: ORA-16751: failed to switchover to physical standby database

 

Failed.

Unable to switchover, primary database is still “db1

 

Below shows the configured DGMGRL

 

DGMGRL> show configuration

Configuration - dg

  Protection Mode: MaxPerformance

  Databases:

    db1 - Primary database

    db2 - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS


Post switch-over both primary and standby have the same database role-- i`.e Primary has physical standby and Standby has also Standby.

Primary DB had been recovered from a failed DGMGRL Broker switch over..  Now Primary DB role is Primary.


The below check from drcdb1.log


2014 19:12:52.360 02001000   359616355 DMON: Entered rfmsoexinst for phase 1

2014-02-14 19:12:52.360                      INSV: Reply received for message with

2014-02-14 19:12:52.360                            req ID 1.1.359616355, opcode CTL_SWITCH, phase BEGIN

2014-02-14 19:12:52.375 02001000   359616355 DMON: posting primary instances for SWITCHOVER phase 2

2014-02-14 19:12:52.375 02001000   359616355 DMON: status from rfi_post_instances() for CTL_SWITCH = ORA-00000

2014-02-14 19:12:52.375 02001000   359616355 DMON: dispersing message to standbys for SWITCHOVER phase TEARDOWN

2014-02-14 19:12:52.375                      INSV: Received message for inter-instance publication

2014-02-14 19:12:52.375                            req ID 1.1.359616355, opcode CTL_SWITCH, phase TEARDOWN, flags 5

2014-02-14 19:12:52.375 02001000   359616355 DMON: Entered rfmsoexinst for phase 2

2014-02-14 19:12:52.375                      RSM0: Received Set State Request: rid=0x01041001, sid=0, phid=1, econd=2, sitehndl=0x02001000

2014-02-14 19:12:52.376                      Log Transport Resource: SetState OFFLINE, phase TEAR-DOWN, External Cond SWITCH-OVER-PHYS_STBY

2014-02-14 19:12:52.376                      RSM0: Received Set State Request: rid=0x01011001, sid=4, phid=1, econd=2, sitehndl=0x02001000

2014-02-14 19:12:52.376                      Database Resource[IAM=PRIMARY]: SetState PHYSICAL-APPLY-ON, phase TEAR-DOWN, External Cond SWITCH-OVER-PHYS_STBY, Target Site Handle 0x02001000

2014-02-14 19:12:52.376                      Executing SQL [ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN]

2014-02-14 19:13:12.345                      RSM0 Notifying DMON of db close

2014-02-14 19:13:23.167                      SQL [ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN] Executed successfully

2014-02-14 19:13:23.199                      Executing SQL [alter system set log_archive_dest_2='']

2014-02-14 19:13:23.234                      SQL [alter system set log_archive_dest_2=''] Executed successfully

2014-02-14 19:13:23.247                      Executing SQL [alter system set log_archive_dest_state_2 = 'ENABLE']

2014-02-14 19:13:23.258                      SQL [alter system set log_archive_dest_state_2 = 'ENABLE'] Executed successfully

2014-02-14 19:13:23.259                      RSM: clearing IncarnationTable internal property of site 0x01010000

2014-02-14 19:13:23.259 02001000   359616355 DMON: Broker determines that instance restart is required:  Operation = SWITCHOVER  status = ORA-16570

2014-02-14 19:13:23.260 02001000   359616355    Resource: db1 (01011001)  State: PHYSICAL-APPLY-ON

2014-02-14 19:13:23.260                      INSV: Reply received for message with

2014-02-14 19:13:23.272                            req ID 1.1.359616355, opcode CTL_SWITCH, phase TEARDOWN

2014-02-14 19:13:23.272 02001000   359616355 DMON: Instance ID 1 returned ORA-16570

2014-02-14 19:13:23.272 02001000   359616355       for phase TEARDOWN of op CTL_SWITCH.

2014-02-14 19:13:23.277 02001000   359616355 DMON: posting primary instances for SWITCHOVER phase 2

2014-02-14 19:13:23.277                      INSV: Received message for inter-instance publication

2014-02-14 19:13:23.277                            req ID 1.1.359616355, opcode CTL_SWITCH, phase TEARDOWN, flags 5

2014-02-14 19:13:23.277 02001000   359616355 DMON: status from rfi_post_instances() for CTL_SWITCH = ORA-00000

2014-02-14 19:13:23.277 02001000   359616355 DMON: dispersing message to standbys for SWITCHOVER phase TEARDOWN

2014-02-14 19:13:23.277 02001000   359616355 DMON: Entered rfmsoexinst for phase 2

2014-02-14 19:13:23.277                      INSV: Reply received for message with

2014-02-14 19:13:23.278                            req ID 1.1.359616355, opcode CTL_SWITCH, phase TEARDOWN

2014-02-14 19:14:04.675                      NSV1: Site db2 returned ORA-16665.

2014-02-14 19:14:24.832                      NSV1: Site db2 returned ORA-16665.

2014-02-14 19:14:46.673                      NSV1: Site db2 returned ORA-16665.

2014-02-14 19:15:08.125                      NSV1: Site db2 returned ORA-16665.

2014-02-14 19:15:29.932                      NSV1: Site db2 returned ORA-16665.

2014-02-14 19:15:54.406                      NSV1: Site db2 returned ORA-16665.

2014-02-14 19:16:16.455                      NSV1: Site db2 returned ORA-16665.

2014-02-14 19:16:38.497                      NSV1: Site db2 returned ORA-16665.

2014-02-14 19:17:00.438                      NSV1: Site db2 returned ORA-16664.

2014-02-14 19:17:00.439 02001000   359616355 DMON: Database db2 returned ORA-16664

2014-02-14 19:17:00.439 02001000   359616355       for opcode = CTL_SWITCH, phase = TEARDOWN, req_id = 1.1.359616355

2014-02-14 19:17:00.440 02001000   359616355 Operation CTL_SWITCH canceled during phase 2, error = ORA-16664

2014-02-14 19:17:00.440 02001000   359616355 DMON: posting primary instances for SWITCHOVER phase 5

2014-02-14 19:17:00.441 02001000   359616355 DMON: status from rfi_post_instances() for CTL_SWITCH = ORA-00000

2014-02-14 19:17:00.441 02001000   359616355 DMON: Switchover Aborted due to errors

2014-02-14 19:17:00.442 02001000   359616355    Site named: db1 is still primary

2014-02-14 19:17:00.442 02001000   359616355    error = ORA-16664

 

The below Check from alert_db1.log

 

Fri Feb 14 19:12:52 2014

ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN

ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY  [Process Id: 4570] (db1)

Fri Feb 14 19:12:56 2014

Thread 1 advanced to log sequence 320 (LGWR switch)

  Current log# 2 seq# 320 mem# 0: /dt1/db1/datafiles/redo2.log

Waiting for all non-current ORLs to be archived...

Waiting for the ORL for thread 1 sequence 319 to be archived...

Fri Feb 14 19:13:01 2014

Archived Log entry 641 added for thread 1 sequence 319 ID 0x5565051a dest 1:

Fri Feb 14 19:13:07 2014

ORL for thread 1 sequence 319 has been archived...

All non-current ORLs have been archived.

Waiting for all FAL entries to be archived...

All FAL entries have been archived.

Waiting for dest_id 2 to become synchronized...

Active, synchronized Physical Standby  switchover target has been identified

Thread 1 cannot allocate new log, sequence 321

Checkpoint not complete

  Current log# 2 seq# 320 mem# 0: /dt1/db1/datafiles/redo2.log

Thread 1 advanced to log sequence 321 (LGWR switch)

  Current log# 1 seq# 321 mem# 0: /dt1/db1/datafiles/redo1.log

ARCH: Standby redo logfile selected for thread 1 sequence 320 for destination LOG_ARCHIVE_DEST_2

Archived Log entry 642 added for thread 1 sequence 320 ID 0x5565051a dest 1:

Fri Feb 14 19:13:08 2014

Fri Feb 14 19:13:08 2014

SMON: disabling tx recovery

Stopping background process QMNC

CLOSE: killing server sessions.

CLOSE: all sessions shutdown successfully.

SMON: disabling cache recovery

Fri Feb 14 19:13:12 2014

Shutting down archive processes

Archiving is disabled

Fri Feb 14 19:13:12 2014

ARCH shutting down

ARC1: Archival stopped

ARC0: Becoming the 'no FAL' ARCH

ARC0: Becoming the 'no SRL' ARCH

ARC0: Archiving disabled

ARCH shutting down

ARC1: Archival stopped

ARC0: Becoming the 'no FAL' ARCH

ARC0: Becoming the 'no SRL' ARCH

ARC0: Archiving disabled

ARCH shutting down

ARC0: Archival stopped

Thread 1 closed at log sequence 321

Successful close of redo thread 1

ARCH: Noswitch archival of thread 1, sequence 321

ARCH: End-Of-Redo Branch archival of thread 1 sequence 321

Archived Log entry 644 added for thread 1 sequence 321 ID 0x5565051a dest 1:

ARCH: Archiving is disabled due to current logfile archival

Primary will check for some target standby to have received all redo

Final check for a synchronized target standby. Check will be made once.

LOG_ARCHIVE_DEST_2 is a potential Physical Standby  switchover target

Active, synchronized target has been identified

Target has also applied all redo

Backup controlfile written to trace file /dt2/diag/rdbms/db1/db1/trace/db1_rsm0_4570.trc

Clearing standby activation ID 1432683802 (0x5565051a)

The primary database controlfile was created using the

'MAXLOGFILES 16' clause.

There is space for up to 14 standby redo logfiles

Use the following SQL commands on the standby database to create

standby redo logfiles that match the primary database:

ALTER DATABASE ADD STANDBY LOGFILE 'srl1.f' SIZE 4194304;

ALTER DATABASE ADD STANDBY LOGFILE 'srl2.f' SIZE 4194304;

ALTER DATABASE ADD STANDBY LOGFILE 'srl3.f' SIZE 4194304;

Archivelog for thread 1 sequence 321 required for standby recovery

Switchover: Primary controlfile converted to standby controlfile succesfully.

Fri Feb 14 19:13:14 2014

MRP0 started with pid=18, OS id=7340

MRP0: Background Managed Standby Recovery process started (db1)

Serial Media Recovery started

Managed Standby Recovery not using Real Time Apply

Online logfile pre-clearing operation disabled by switchover

Media Recovery Log /arch/1_321_838858074.arc

Identified End-Of-Redo for thread 1 sequence 321

Resetting standby activation ID 0 (0x0)

Media Recovery End-Of-Redo indicator encountered

Media Recovery Applied until change 559881

MRP0: Media Recovery Complete: End-Of-REDO (db1)

MRP0: Background Media Recovery process shutdown (db1)

Fri Feb 14 19:13:23 2014

Switchover: Complete - Database shutdown required (db1)

Completed: ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN

Using STANDBY_ARCHIVE_DEST parameter default value as /arch

ALTER SYSTEM SET log_archive_dest_2='' SCOPE=BOTH;

Using STANDBY_ARCHIVE_DEST parameter default value as /arch

ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=BOTH;

 

Fri Feb 14 21:09:42 2014

Starting ORACLE instance (normal)

LICENSE_MAX_SESSION = 0

LICENSE_SESSIONS_WARNING = 0

Picked latch-free SCN scheme 2

Autotune of undo retention is turned on.

IMODE=BR

ILAT =167

LICENSE_MAX_USERS = 0

----------------------------------------------------------------


Regards,

  • 1. Re: Switch Over Failing through DGMGRL
    user591655 Journeyer
    Currently Being Moderated

    2014-02-14 19:17:00.439 02001000   359616355 DMON: Database db2 returned ORA-16664

    2014-02-14 19:17:00.439 02001000   359616355       for opcode = CTL_SWITCH, phase = TEARDOWN, req_id = 1.1.359616355

    2014-02-14 19:17:00.440 02001000   359616355 Operation CTL_SWITCH canceled during phase 2, error = ORA-16664

    2014-02-14 19:17:00.440 02001000   359616355 DMON: posting primary instances for SWITCHOVER phase 5

    2014-02-14 19:17:00.441 02001000   359616355 DMON: status from rfi_post_instances() for CTL_SWITCH = ORA-00000

    2014-02-14 19:17:00.441 02001000   359616355 DMON: Switchover Aborted due to errors

    2014-02-14 19:17:00.442 02001000   359616355    Site named: db1 is still primary

    2014-02-14 19:17:00.442 02001000   359616355    error = ORA-16664


    ora-16664 happens when there is a connection problem between the primary and the standby database. You need to check the listener and tnsnames files and ensure that there is no connection problem between the primary and the standby database.

  • 2. Re: Switch Over Failing through DGMGRL
    990991 Newbie
    Currently Being Moderated

    Hi..

    if there`s no connection.. or if that is connection problem then it wouldn`t have shown the primary and standby are in sync and at the same time, it wouldn`t even let me switch manually (Primary to standby and vice versa)... i checked manual switching..  it was working fine at that time.. and i checked removing configuration... and reconfigured DGMGRL.. but the same error persisting...

  • 3. Re: Switch Over Failing through DGMGRL
    990991 Newbie
    Currently Being Moderated

    I tried switching over through dgmgrl again.. standby has become primary and primary has become standby.. when checked database role.. but dgmgrl still throws the same error...

     

    DGMGRL> switchover to db2

    Performing switchover NOW, please wait...


    Performing switchover NOW, please wait…

    Error: ORA-16751: failed to switchover to physical standby database

     

    Failed.

    Unable to switchover, primary database is still “db1

     

    Regards,

  • 4. Re: Switch Over Failing through DGMGRL
    Tom321 Journeyer
    Currently Being Moderated

    Hi,

     

    during failover the primary is shutdown and the regular tnsnames entry therefore can't be resolved by the listener. For switchover broker needs a static entry at the listener to complete the failover. Please check you have similar entries on both listeners:

     

    listener.ora:

    ...

    SID_LIST_LISTENER =

    (SID_LIST =

      (SID_DESC =

       (GLOBAL_DBNAME = db1_DGMGRL)

       (ORACLE_HOME = /oracle/orabase/ora11g)

       (SID_NAME = db1)

      )

    ...

     

    Check the broker config with 'show database verbose db1', the parameter StaticConnectIdentifier shows you what broker expects to find for your configuration.

    Also make sure to connect with "connect sys@dg1' in dgmgrl for the switchover, do not use OS authentication ('connect target /').

     

    Regards

    Thomas


Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points