If you use Data Guard, you probably use its broker.  You may not have noticed, but whenever you make changes to your Broker configuration, those changes are stored in a file on the operating system.  The location of that file is specified in the DG_BROKER_CONFIG_FILE1 initialisation parameter.  DG_BROKER_CONFIG_FILE2 is a backup of DG_BROKER_CONFIG_FILE1, as we are going to see in this post.  Note that, as the Oracle documentation says, these initialisation parameters can be modified only when the broker is not running.

Let's look at what happens to those files when you create/delete/re-create/modify a broker configuration in Oracle 12cR1.

 

Here is the situation at 11H30 on the 29th of May (Database MYDB4 is on server machine2, and database MYDB3 is on server machine1):

machine2@oracle::~ # ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 12K May 29 10:02 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

-rw-rw---- 1 oracle oracle 12K May 29 10:02 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

machine1@oracle::~ # ls -rtlh /app/machine1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 12K May 29 10:03 /app/racnode1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

-rw-rw---- 1 oracle oracle 12K May 29 11:22 /app/racnode1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

 

Let's do away with the current broker configuration:

DGMGRL> remove configuration

Warning: ORA-16620: one or more databases could not be reached for a delete operation

ORA-16620 occurs when the broker is not started on either of the databaseses.  in that case, the broker configuration files are not affected.  Let's start the broker on our STANDBY and let's do that again.

DGMGRL> remove configuration

Removed configuration

 

Now let's create a new broker configuration:

DGMGRL> create configuration MYDB3dgconf as primary database is 'MYDB3' connect identifier is MYDB3_DGMGRL;

Configuration "MYDB3dgconf" created with primary database "MYDB3"

 

Since I am working on machine1, the broker configuration files on machine1 have been affected at 11:35AM:

racnode1@oracle::~ # ls -rtlh /app/machine1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 8.0K May 29 11:35 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

-rw-rw---- 1 oracle oracle  12K May 29 11:35 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

 

Now let's enable that configuration:

DGMGRL>  enable configuration

Enabled.

 

machine1@oracle::~ # ls -rtlh /app/machine1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 12K May 29 11:35 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

-rw-rw---- 1 oracle oracle 12K May 29 11:36 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

As you can see, only one of the configuration files was modified (at 11:36AM).  This is because the other one is a backup of the previous state of the configuration (when it was not enabled).

 

At 11:38AM:

DGMGRL> add database 'MYDB4' as connect identifier is MYDB4_DGMGRL maintained as physical;

Error: ORA-16525: The Oracle Data Guard broker is not yet available.

Failed.

 

racnode1@oracle::~ # ls -rtlh /app/machine1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 12K May 29 11:35 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

-rw-rw---- 1 oracle oracle 12K May 29 11:36 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

No files were modified.  When you generate an error in DGMGRL, the broker configuration files are not affected.

Here is the explanation in the Data Guard log on racnode1:

ADD DATABASE

      05/29/2018 11:38:54

      DG_BROKER_START is not set to TRUE on member MYDB4. Set the parameter to TRUE and then try to add the member to the configuration.

 

So let's correct that:

29-05 11:43 MYDB4 SYS AS SYSDBA> alter system set dg_broker_start=TRUE;

System altered.

 

Now at 11:45AM:

DGMGRL> add database 'MYDB4' as connect identifier is MYDB4_DGMGRL maintained as physical;

Database "MYDB4" added

 

racnode1@oracle::~ # ls -rtlh /app/machine1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 12K May 29 11:36 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

-rw-rw---- 1 oracle oracle 12K May 29 11:45 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

On machine1, one configuration file was altered, and the other is a backup of the previous state.

But on racnode2, both configuration files were modified (in fact, re-created), since this is the first time this new broker configuration is generating a change on racnode2 (remember, that broker configuration was created at 11:35AM on machine1 and so far had not done anything involving racnode2).

machine2@oracle::~ # ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 8.0K May 29 11:44 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

-rw-rw---- 1 oracle oracle 8.0K May 29 11:44 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

 

But the STANDBY broker log reports an error:

05/29/2018 11:44:50

      Site MYDB3 returned ORA-16603.

      MYDB4 version check failed; status = ORA-16603: detected configuration unique ID mismatch

            configuration will be deleted on this database

 

Do you remember that the broker of my STANDBY was not started when I ran CREATE CONFIGURATION on the PRIMARY?  So let's re-create the broker configuration while both brokers are started.

 

At 11:55AM:

DGMGRL> remove configuration

Warning: ORA-16620: one or more databases could not be reached for a delete operation

Removed configuration

 

machine1@oracle::~ # ls -rtlh /app/machine1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 8.0K May 29 11:55 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

-rw-rw---- 1 oracle oracle 8.0K May 29 11:55 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

As you can see, when you remove a broker configuration, there is no backup of the removed configuration since both files are reset (their size is 8K, while when there was a configuration, their size was 12K).

At that point, I had to move to another subject, and my colleague finished that broker configuration.  But on the following day (the 30th of May), I had to re-create it.

Here is the situation on the morning of May30:

machine1@oracle:MYDB3:/app/machine1/oracle/adm/MYDB3 #  ls -rtlh /app/racnode1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 12K May 29 21:01 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

-rw-rw---- 1 oracle oracle 12K May 29 21:01 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

 

At 10:40AM, I removed the broker configuration from racnode1:

DGMGRL>  remove configuration;

Removed configuration

 

machine1@oracle:MYDB3:/app/machine1/oracle/adm/MYDB3 #  ls -rtlh /app/racnode1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 8.0K May 30 10:41 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

-rw-rw---- 1 oracle oracle 8.0K May 30 10:41 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

machine2@oracle::~ #  ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 8.0K May 30 10:40 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

-rw-rw---- 1 oracle oracle 8.0K May 30 10:40 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

It seems it first connects to the other server (the files on that one were reset at 10:40, while the files on machine1 were reset 1 minute later).

 

Now let's create a new broker congiration, with MYDB3 as the PRIMARY:

DGMGRL> create configuration MYDB3dgconf as primary database is 'MYDB3' connect identifier is MYDB3_DGMGRL;

Error: ORA-16584: operation cannot be performed on a standby database

Failed.

 

So let's go to the other server (machine2):

DGMGRL for Linux: Version 12.1.0.2.0 - 64bit Production

Copyright (c) 2000, 2013, Oracle. All rights reserved.

Welcome to DGMGRL, type "help" for information.

DGMGRL> connect sys

Password:

Connected as SYSDG.

 

DGMGRL> create configuration MYDB3dgconf as primary database is 'MYDB3' connect identifier is MYDB3_DGMGRL;

Error: ORA-16642: DB_UNIQUE_NAME mismatch

Failed.

This error occured at 10:44AM, and it did modify the configuration files:

machine2@oracle::~ #  ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 8.0K May 30 10:44 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

-rw-rw---- 1 oracle oracle 8.0K May 30 10:44 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

But not on the STANDBY side (machine1):

machine1@oracle:MYDB3:/app/machine1/oracle/adm/MYDB3 #  ls -rtlh /app/machine1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 8.0K May 30 10:41 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

-rw-rw---- 1 oracle oracle 8.0K May 30 10:41 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

 

Let's create our configuration but with the right PRIMARY this time:

DGMGRL> create configuration MYDB4dgconf as primary database is 'MYDB4' connect identifier is MYDB4_DGMGRL;

Configuration "MYDB4dgconf" created with primary database "MYDB4"

 

As seen previously, when you create a configuration, both files are affected (but notice their difference in size):

machine2@oracle::~ #  ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 8.0K May 30 10:46 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

-rw-rw---- 1 oracle oracle  12K May 30 10:46 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

But on the STANDBY side, nothing happened, as the STANDBY has not been added to the configuration yet:

machine1@oracle:MYDB3:/app/machine1/oracle/adm/MYDB3 #  ls -rtlh /app/machine1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 8.0K May 30 10:41 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

-rw-rw---- 1 oracle oracle 8.0K May 30 10:41 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

 

Let's enable that configuration:

DGMGRL> enable configuration

Enabled.

 

machine2@oracle::~ #  ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 12K May 30 10:46 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

-rw-rw---- 1 oracle oracle 12K May 30 10:48 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

Again, one of the files contains the previous state of the configuration (2mn earlier, before the enablement).

 

Now, let's add a STANDBY to that configuration:

DGMGRL> add database 'MYDB3' as connect identifier is MYDB3_DGMGRL maintained as physical;

Error: ORA-16698: LOG_ARCHIVE_DEST_n parameter set for object to be added

Failed.

machine2@oracle::~ #  ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 12K May 30 10:46 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

-rw-rw---- 1 oracle oracle 12K May 30 10:48 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

That error did not affect the configuration files.

 

Before we modify initialisation parameter log_archive_dest_3 (which is the cause of the ORA-16698), we must remove our configuration.  We will re-create it after that modification in our database.

DGMGRL> remove configuration;

Removed configuration

 

machine2@oracle::~ #  ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 8.0K May 30 11:00 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

-rw-rw---- 1 oracle oracle 8.0K May 30 11:00 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

Of course, all files are reset.

Now let's modify that parameter in both databases:

alter system set log_archive_dest_3='' scope =both ;

 

And re-create our broker configuration:

DGMGRL> create configuration MYDB4dgconf as primary database is 'MYDB4' connect identifier is MYDB4_DGMGRL;

Configuration "MYDB4dgconf" created with primary database "MYDB4"

 

machine2@oracle::~ #  ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 8.0K May 30 11:48 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

-rw-rw---- 1 oracle oracle  12K May 30 11:48 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

One file on machine2 contains our new configuration while the other one is a backup of the previous state (no configuration, hence 8K).  While, on machine1, nothing happened, but you already know that.

 

DGMGRL> enable configuration;

Enabled.

 

machine2@oracle::~ #  ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 12K May 30 11:48 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

-rw-rw---- 1 oracle oracle 12K May 30 11:50 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

One file on machine2 contains our new configuration (the one dated 11:50AM) while the other one is a backup of the previous state.

 

At 11:52AM:

GMGRL>  add database 'MYDB3' as connect identifier is MYDB3_DGMGRL maintained as physical;

Database "MYDB3" added

 

-rw-rw---- 1 oracle oracle 12K May 30 11:50 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

machine2@oracle::~ #  ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 12K May 30 11:50 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

-rw-rw---- 1 oracle oracle 12K May 30 11:52 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

machine1@oracle:MYDB3:/app/machine1/oracle/adm/MYDB3 #  ls -rtlh /app/machine1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 8.0K May 30 10:41 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

-rw-rw---- 1 oracle oracle 8.0K May 30 10:41 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

The files on the STANDBY side have not yet been modifie, but you already know why.

 

At 12:02PM:

DGMGRL> enable database 'MYDB3';

Enabled.

 

machine2@oracle::~ #  ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 12K May 30 12:02 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

-rw-rw---- 1 oracle oracle 12K May 30 12:02 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

machine1@oracle:MYDB3:/app/machine1/oracle/adm/MYDB3 #  ls -rtlh /app/machine1/oracle/product/12.1.0/dbs/dr*3*

-rw-rw---- 1 oracle oracle 12K May 30 12:02 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

-rw-rw---- 1 oracle oracle 12K May 30 12:03 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

At last, the files on the STANDBY side are modified.

And in my PRIMARY, this ENABLE DATABASE has automatically filled in the log_archive_dest_3 parameter with "service="MYDB3_dgmgrl", ASYNC NOAFFIRM delay=0 optional compression=disable max_failure=0 max_connections=1 reopen=300 db_unique_name="MYDB3" net_timeout=30, valid_for=(online_logfile,all_roles)".

And in my STANDBY, this ENABLE DATABASE has automatically filled in the fal_server parameter "with MYDB4_dgmgrl" and the log_archive_config parameter has changed from "nodg_config" to "dg_config=(MYDB4)".

 

If I go to machine1 and run SHOW CONFIGURATION, everything is OK:

DGMGRL> show configuration

Configuration - MYDB4dgconf

  Protection Mode: MaxPerformance

  Members:

  MYDB4 - Primary database

    MYDB3 - Physical standby database

Fast-Start Failover: DISABLED

Configuration Status:

SUCCESS   (status updated 79 seconds ago)

 

Let's now perform a SWITCHOVER (after first running VALIDATE DATABASE for both our databases, to confirm the SWITCHOVER can go through):

DGMGRL> switchover to 'MYDB3' ;

Performing switchover NOW, please wait...

Operation requires a connection to instance "MYDB3" on database "MYDB3"

Connecting to instance "MYDB3"...

Connected as SYSDBA.

New primary database "MYDB3" is opening...

Operation requires start up of instance "MYDB4" on database "MYDB4"

Starting instance "MYDB4"...

ORACLE instance started.

Database mounted.

Switchover succeeded, new primary is "MYDB3"

 

-rw-rw---- 1 oracle oracle 12K May 30 13:22 /app/machine1/oracle/product/12.1.0/dbs/dr2MYDB3.dat

-rw-rw---- 1 oracle oracle 12K May 30 13:22 /app/machine1/oracle/product/12.1.0/dbs/dr1MYDB3.dat

machine2@oracle:MYDB4:/app/machine2/oracle/product/12.1.0/network/admin #  ls -rtlh /app/machine2/oracle/product/12.1.0/dbs/dr*4*

-rw-rw---- 1 oracle oracle 12K May 30 13:21 /app/machine2/oracle/product/12.1.0/dbs/dr2MYDB4.dat

-rw-rw---- 1 oracle oracle 12K May 30 13:21 /app/machine2/oracle/product/12.1.0/dbs/dr1MYDB4.dat

Again, the configuration files on the other server were modified first.  But all 4 files have been modified, and 2 of them (one on either machines) contain the previous state of the broker configuration.