10 Replies Latest reply: Dec 20, 2012 3:59 PM by user275290 RSS

    Error: ORA-16525: the Data Guard broker is not yet available

    user275290
      Hi ,

      After upgrading from 11201 to 11203 ON AIX GI/RDBMS on standby but have not upgraded the primary db yet.I had set dg_broker_start=false and disable configuration before i started the upgrade .

      once the GI for oracle restart was upgraded i upgraded the rdbms binaries and brought up the standby on mount ,while trying to enable configuration its throwing the below error.I had already started the broker process.

      SQL> show parameter dg_

      NAME TYPE VALUE
      ------------------------------------ ----------- ------------------------------
      dg_broker_config_file1 string /u01/app/omvmxp1/product/11.2.
      0/dbhome_2/dbs/dr1mvmxs2.dat
      dg_broker_config_file2 string /u01/app/omvmxp1/product/11.2.
      0/dbhome_2/dbs/dr2mvmxs2.dat
      dg_broker_start boolean TRUE


      DGMGRL> show configuration;

      Configuration - Matrxrep_brkr

      Protection Mode: MaxAvailability
      Databases:
      mvmxp2 - Primary database
      mvmxs2 - Physical standby database
      Error: ORA-16525: the Data Guard broker is not yet available

      Fast-Start Failover: DISABLED

      Configuration Status:
      ERROR

      from drcmvmxs2.log
      Starting Data Guard Broker bootstrap <<
      Broker Configuration File Locations:
      dg_broker_config_file1 = "/u01/app/omvmxp1/product/11.2.0/dbhome_2/dbs/dr1mvmxs2.dat"
      dg_broker_config_file2 = "/u01/app/omvmxp1/product/11.2.0/dbhome_2/dbs/dr2mvmxs2.dat"
      12/19/2012 16:05:33
      Data Guard Broker shutting down
      DMON Process Shutdown <<
      12/19/2012 16:10:20
      Starting Data Guard Broker bootstrap <<
      Broker Configuration File Locations:
      dg_broker_config_file1 = "/u01/app/omvmxp1/product/11.2.0/dbhome_2/dbs/dr1mvmxs2.dat"
      dg_broker_config_file2 = "/u01/app/omvmxp1/product/11.2.0/dbhome_2/dbs/dr2mvmxs2.dat"
      ~



      Regards

      Edited by: Monto on Dec 19, 2012 1:23 PM
        • 1. Re: Error: ORA-16525: the Data Guard broker is not yet available
          mseberg
          Hello;

          Wait about 3 minutes and try the command again, then post the results

          H2. Never mind, I'm thinking ORA-16610.

          Let me double check my notes

          later

          Would check

          LocalListenerAddress property and change if incorrect and then

          SHOW CONFIGURATION VERBOSE;


          Best Regards

          mseberg

          Edited by: mseberg on Dec 19, 2012 3:29 PM

          Edited by: mseberg on Dec 19, 2012 3:32 PM
          • 2. Re: Error: ORA-16525: the Data Guard broker is not yet available
            user275290
            Nope no change same error.

            DGMGRL> show configuration;

            Configuration - Matrxrep_brkr

            Protection Mode: MaxAvailability
            Databases:
            mvmxp2 - Primary database
            mvmxs2 - Physical standby database
            Error: ORA-16525: the Data Guard broker is not yet available

            Fast-Start Failover: DISABLED

            Configuration Status:
            ERROR


            DGMGRL> show configuration verbose;

            Configuration - Matrxrep_brkr

            Protection Mode: MaxAvailability
            Databases:
            mvmxp2 - Primary database
            mvmxs2 - Physical standby database
            Error: ORA-16525: the Data Guard broker is not yet available

            Properties:
            MIV = '88'
            FastStartFailoverThreshold = '30'
            PRIMARY_SITE_ID = '513'
            HEALTH_CHECK_INTERVAL = '6000'
            DRC_UNIQUE_ID = '12251'
            DRC_UNIQUE_ID_SEQUENCE = '2'
            OVERALL_PROTECTION_MODE = '2'
            FastStartFailoverLagLimit = '30'
            CommunicationTimeout = '180'
            FastStartFailoverAutoReinstate = 'TRUE'
            FastStartFailoverPmyShutdown = 'TRUE'
            ObserverInfo = ''
            BystandersFollowRoleChange = 'ALL'

            Fast-Start Failover: DISABLED

            Configuration Status:
            ERROR
            SQL> show parameter local_

            NAME TYPE VALUE
            ------------------------------------ ----------- ------------------------------
            local_listener string (DESCRIPTION=(ADDRESS_LIST=(AD
            DRESS=(PROTOCOL=TCP)(HOST=xxxxxx)(PORT=105
            25))))
            log_archive_local_first boolean TRUE

            Thanks

            Edited by: Monto on Dec 19, 2012 1:36 PM
            • 3. Re: Error: ORA-16525: the Data Guard broker is not yet available
              CKPT
              Check the broker log files and post last 50-70 lines of information of it?
              Note:= Stop and start the broker before capturing log .

              Edited by: CKPT on Dec 20, 2012 4:31 AM
              • 4. Re: Error: ORA-16525: the Data Guard broker is not yet available
                mseberg
                My friend CKPT wrote the book on this.

                If you let him help you, your issue will get solved.

                Assume you know the location of the broker logs.

                Best Regards

                mseberg
                • 5. Re: Error: ORA-16525: the Data Guard broker is not yet available
                  user275290
                  Hi,

                  I removed the configuration and removed the broker files from RAC primary(mvmxp2) and single instance standby(mvmxs2) and re-created back.i tried it many times but getting error "ORA-16532" .I needed to have this standby backup before i start upgrading the primary.

                  SQL> alter system set dg_broker_start=true scope=both;

                  System altered.

                  SQL> exit
                  Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
                  With the Partitioning, Automatic Storage Management, OLAP, Data Mining
                  and Real Application Testing options
                  palmer60:/u01/app/omvmxp1/product/11.2.0/dbhome_2/dbs>dgmgrl
                  DGMGRL for IBM/AIX RISC System/6000: Version 11.2.0.3.0 - 64bit Production

                  Copyright (c) 2000, 2009, Oracle. All rights reserved.

                  Welcome to DGMGRL, type "help" for information.
                  DGMGRL> connect sys@mvmxp2
                  Password:
                  Connected.
                  DGMGRL> CREATE CONFIGURATION 'Matrxrep'
                  AS
                  PRIMARY DATABASE IS 'mvmxp2'
                  CONNECT IDENTIFIER IS 'mvmxp2';> > >
                  Configuration "Matrxrep" created with primary database "mvmxp2"
                  DGMGRL> ADD DATABASE 'mvmxs2'
                  AS
                  CONNECT IDENTIFIER IS 'mvmxs2'
                  ;
                  Database "mvmxs2" added
                  DGMGRL> SHOW CONFIGURATION;

                  Configuration - Matrxrep

                  Protection Mode: MaxPerformance
                  Databases:
                  mvmxp2 - Primary database
                  mvmxs2 - Physical standby database

                  Fast-Start Failover: DISABLED

                  Configuration Status:
                  DISABLED

                  DGMGRL> ENABLE CONFIGURATION;
                  Enabled.
                  DGMGRL> SHOW DATABASE MVMXS2;

                  Database - mvmxs2

                  Role: PHYSICAL STANDBY
                  Intended State: APPLY-ON
                  Transport Lag: (unknown)
                  Apply Lag: (unknown)
                  Real Time Query: OFF
                  Instance(s):
                  mvmxs2

                  Database Status:
                  DGM-17016: failed to retrieve status for database "mvmxs2"
                  ORA-16532: Data Guard broker configuration does not exist
                  ORA-16625: cannot reach database "mvmxs2"

                  DGMGRL>

                  tailed the drcmvmxs2.log during stop and start of the broker

                  palmer60:/u01/app/omvmxp1/diag/rdbms/mvmxs2/mvmxs2/trace>tail -f drcmvmxs2.log
                  12/19/2012 20:32:20
                  drcx: cannot open configuration file "/u01/app/omvmxp1/product/11.2.0/dbhome_2/dbs/dr1mvmxs2.dat"
                  ORA-27037: unable to obtain file status
                  IBM AIX RISC System/6000 Error: 2: No such file or directory
                  Additional information: 3
                  12/19/2012 20:32:55
                  drcx: cannot open configuration file "/u01/app/omvmxp1/product/11.2.0/dbhome_2/dbs/dr2mvmxs2.dat"
                  ORA-27037: unable to obtain file status
                  IBM AIX RISC System/6000 Error: 2: No such file or directory
                  Additional information: 3
                  12/19/2012 20:59:10
                  Data Guard Broker shutting down
                  DMON Process Shutdown <<
                  12/19/2012 20:59:35
                  Starting Data Guard Broker bootstrap <<
                  Broker Configuration File Locations:
                  dg_broker_config_file1 = "/u01/app/omvmxp1/product/11.2.0/dbhome_2/dbs/dr1mvmxs2.dat"
                  dg_broker_config_file2 = "/u01/app/omvmxp1/product/11.2.0/dbhome_2/dbs/dr2mvmxs2.dat"

                  Not sure how to fix this one.

                  Regards
                  • 6. Re: Error: ORA-16525: the Data Guard broker is not yet available
                    Mahir M. Quluzade
                    Hi Monto!

                    Can you check tnsnames.ora on standby database server, both databases , primary and standby database added to tnsnames.ora ?
                    Can you paste here tnsnames.ora content from standby?


                    And can you paste here following scripts results?

                    On both server
                     show parameter dg_broker
                    Regards
                    Mahir M. Quluzade

                    Edited by: Mahir M. Quluzade on Dec 20, 2012 11:26 AM
                    • 7. Re: Error: ORA-16525: the Data Guard broker is not yet available
                      CKPT
                      >
                      palmer60:/u01/app/omvmxp1/diag/rdbms/mvmxs2/mvmxs2/trace>tail -f drcmvmxs2.log
                      12/19/2012 20:32:20
                      drcx: cannot open configuration file "/u01/app/omvmxp1/product/11.2.0/dbhome_2/dbs/dr1mvmxs2.dat"
                      ORA-27037: unable to obtain file status
                      IBM AIX RISC System/6000 Error: 2: No such file or directory
                      >

                      I think the above information is enough to look at it.
                      The broker file unable to open by the process and hence failed to read the configuration repository information from the broker config file.
                      Also crosscheck with the permissions on broker config files.

                      And you can delete those files on primary and standby databases, Then start fresh configuration of Broker. It will take minutes to configure. :)

                      1) SQL> alter system set dg_broker_start=TRUE (both primary and standby)
                      2) Connect to DGMGRL on primary
                      3) DGMGRL> CREATE CONFIGURATION 'Confi_name' AS PRIMARY DATABASE IS 'DB_UNIQUE_NAME' CONNECT IDENTIFIER IS <tns string>;
                      4) DGMGRL> ADD DATABASE 'DB_UNIQUE_NAME' AS CONNECT IDENTIFIER IS '<tns string>';
                      5) DGMGRL> enable configuration;
                      6) DGMGRL> show configuraiton;

                      Hope this helps.
                      • 8. Re: Error: ORA-16525: the Data Guard broker is not yet available
                        user275290
                        I tried re-creating the broker if you see my last post carefully but still getting the errors.The broker configuration files exists on standby as shown below .
                        palmer60:/home/omvmxp1>ls -ltr /u01/app/omvmxp1/product/11.2.0/dbhome_2/dbs/
                        total 144
                        -rw-r--r--    1 omvmxp1  oinstall       2851 May 15 2009  init.ora
                        -rw-r-----    1 omvmxp1  oinstall       5632 Dec 19 15:02 spfilemvmxs2.ora_021412
                        -rw-r-----    1 omvmxp1  oinstall       1536 Dec 19 15:02 orapwmvmxs2
                        -rw-r--r--    1 omvmxp1  oinstall         83 Dec 19 15:02 initmvmxs2.ora
                        -rwxr-xr-x    1 omvmxp1  oinstall       1373 Dec 19 15:02 initmvmxs1.ora
                        -rw-r-----    1 omvmxp1  oinstall         24 Dec 19 15:22 lkMVMXS2
                        -rw-r-----    1 omvmxp1  asmadmin      20480 Dec 19 20:59 dr2mvmxs2.dat
                        -rw-r-----    1 omvmxp1  asmadmin      20480 Dec 19 22:00 dr1mvmxs2.dat
                        -rw-rw----    1 omvmxp1  oinstall       1544 Dec 20 10:08 hc_mvmxs2.dat
                        palmer60:/home/omvmxp1>
                        Regards
                        • 9. Re: Error: ORA-16525: the Data Guard broker is not yet available
                          CKPT
                          >
                          -rw-r----- 1 omvmxp1 asmadmin 20480 Dec 19 20:59 dr2mvmxs2.dat
                          -rw-r----- 1 omvmxp1 asmadmin 20480 Dec 19 22:00 dr1mvmxs2.dat
                          >

                          See the owner of these files and groups of these files..
                          It should be Oracle user, see as below from my environment, I already mentioned in my previous posts. Change ownership
                          Example:
                          $chown oracle:dba dr2mvmxs2.dat
                          $chown oracle:dba dr1mvmxs2.dat
                          oracle-ckpt.com> ls -ltr dr*
                          -rw-r----- 1 oracle oinstall 16384 Dec  8 23:58 dr2ckpt.dat
                          -rw-r----- 1 oracle oinstall 16384 Dec  8 23:59 dr1ckpt.dat
                          oracle-ckpt.com>
                          Hope this clears.
                          • 10. Re: Error: ORA-16525: the Data Guard broker is not yet available
                            user275290
                            I'll get back on this once my GI upgrade on primary is done.I'm stuck with GI rootupgrade.sh failing on 2nd node.

                            Thanks