3 Replies Latest reply on Jun 10, 2016 2:46 PM by HristiyanK

    Errors during Databaguard broker switchover

    Christophe Lize

      Hi,

       

      Env: Oracle 11.2.0.4 EE latest bundle patch on Windows Server 2012R2.

       

      I just setup a primary/standby pair managed by Dataguard Broker and when I switchover, everything is OK except that I have few of the following errors on the standby server (which is becomming the primary) :

       

       

      Fatal NI connect error 12518, connecting to:

      (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=SRV10-W2012)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=dbstatst)(CID=(PROGRAM=c:\app\oracle\product\11.2.0.4\dbhome_1\bin\ORACLE.EXE)(HOST=SRV7-W2012)(USER=SYSTEM))))

       

       

        VERSION INFORMATION:

        TNS for 64-bit Windows: Version 11.2.0.4.0 - Production

        Windows NT TCP/IP NT Protocol Adapter for 64-bit Windows: Version 11.2.0.4.0 - Production

        Time: 12-MAY-2016 14:09:03

        Tracing not turned on.

        Tns error struct:

          ns main err code: 12564

         

      TNS-12564: TNS:connection refused

          ns secondary err code: 0

          nt main err code: 0

          nt secondary err code: 0

          nt OS err code: 0

      Error 12518 received logging on to the standby

      PING[ARC0]: Heartbeat failed to connect to standby 'dbstatst'. Error is 12518.

       

      This is happening during the reboot of the primary database which is switching to standby role. To me, this normal to me and I know that it can be ignored but is there a way to avoid these errors. Anything I should stop before switching over.

       

      I looked at note : 11.2 Data Guard Physical Standby Switchover Best Practices using the Broker (Doc ID 1305019.1)

      but found nothing related to that...

       

      Any idea?

       

      Thanks

       

      Christophe

        • 1. Re: Errors during Databaguard broker switchover
          HristiyanK

          Hello,

           

          What do you mean by "everything is OK" ? Error suddenly stops? Databases are in sync with new roles? Can you check the broker log file? I should be created in the same directory as the alert log and is named drc<$ORACLE_SID>.log. You may check https://docs.oracle.com/cd/E11882_01/server.112/e40771/troubleshooting.htm#DGBKR1060 if you haven't already.

           

          Regards,

          Hristiyan

          • 2. Re: Re: Errors during Databaguard broker switchover
            Christophe Lize

            Hi Hristiyan,

             

            Thank you for answering....

             

            Here is what happen during switchover:

            DGMGRL> show configuration;

             

            Configuration - DBSTATER_DBSTATST

             

              Protection Mode: MaxPerformance

              Databases:

                dbstater - Primary database

                dbstatst - Physical standby database

             

            Fast-Start Failover: DISABLED

             

            Configuration Status:

            SUCCESS

             

            DGMGRL> switchover to "dbstatst";

            Performing switchover NOW, please wait...

            Operation requires a connection to instance "dbstatst" on database "dbstatst"

            Connecting to instance "dbstatst"...

            Connected.

            New primary database "dbstatst" is opening...

            Operation requires startup of instance "dbstater" on database "dbstater"

            Starting instance "dbstater"...

            ORACLE instance started.

            Database mounted.

            Switchover succeeded, new primary is "dbstatst"


            DGMGRL> show configuration;

             

            Configuration - DBSTATER_DBSTATST

             

              Protection Mode: MaxPerformance

              Databases:

                dbstatst - Primary database

                dbstater - Physical standby database

             

            Fast-Start Failover: DISABLED

             

            Configuration Status:

            SUCCESS

             

            drcdbstater.log:

            06/10/2016 09:26:33

            SWITCHOVER TO dbstatst

            Command SWITCHOVER TO dbstatst completed with warning ORA-16523

            SWITCHOVER TO dbstatst

            Notifying Oracle Clusterware to teardown primary database for SWITCHOVER

            06/10/2016 09:26:38

            Command SWITCHOVER TO dbstatst completed

            Shutting down instance after CTL_SWITCH

            Creating Data Guard Broker Monitor Process (DMON)

            06/10/2016 09:27:00

            >> Starting Data Guard Broker bootstrap <<

            Broker Configuration File Locations:

                  dg_broker_config_file1 = "C:\APP\ORACLE\PRODUCT\11.2.0.4\DBHOME_1\DATABASE\DR1DBSTATER.DAT"

                  dg_broker_config_file2 = "C:\APP\ORACLE\PRODUCT\11.2.0.4\DBHOME_1\DATABASE\DR2DBSTATER.DAT"

            06/10/2016 09:27:03

            DMON Registering service DBSTATER_DGB with listener(s)

            Broker Configuration:       "DBSTATER_DBSTATST"

                  Protection Mode:            Maximum Performance

                  Fast-Start Failover (FSFO): Disabled, flags=0x0, version=0

                  Primary Database:           dbstatst (0x02010000)

                  Standby Database:           dbstater, Enabled Physical Standby (0x01010000)

            06/10/2016 09:27:07

            Creating process RSM0

            06/10/2016 09:27:16

            Notifying Oracle Clusterware to buildup standby database after SWITCHOVER

            Command SWITCHOVER TO dbstater completed

             

            drcdbstatst.log

            06/10/2016 09:26:33

            Notifying Oracle Clusterware to teardown target standby database for SWITCHOVER

            06/10/2016 09:26:39

            Command SWITCHOVER TO dbstatst completed

            06/10/2016 09:26:44

            Deferring associated archivelog destinations of sites permanently disabled due to Switchover

            Notifying Oracle Clusterware to buildup primary database after SWITCHOVER

            Command SWITCHOVER TO dbstatst completed

            06/10/2016 09:26:52

            Redo transport problem detected: redo transport for database dbstater has the following error:

              ORA-12518: TNS:listener could not hand off client connection

            Site dbstater returned ORA-16525.

            Data Guard Broker Status Summary:

              Type                        Name                             Severity  Status

              Configuration               DBSTATER_DBSTATST                 Warning  ORA-16607

              Primary Database            dbstatst                            Error  ORA-16778

              Physical Standby Database   dbstater                            Error  ORA-16525

            06/10/2016 09:27:16

            Deferring associated archivelog destinations of sites permanently disabled due to Switchover

            Notifying Oracle Clusterware to buildup primary database after SWITCHOVER

            Command SWITCHOVER TO dbstater completed

             

            listener.ora on primary server :

            # listener.ora Network Configuration File: C:\app\oracle\product\11.2.0.4\dbhome_1\network\admin\listener.ora

            # Generated by Oracle configuration tools.

             

            SID_LIST_LISTENER =

              (SID_LIST =

                (SID_DESC =

                  (SID_NAME = CLRExtProc)

                  (ORACLE_HOME = C:\app\oracle\product\11.2.0.4\dbhome_1)

                  (PROGRAM = extproc)

                  (ENVS = "EXTPROC_DLLS=ONLY:C:\app\oracle\product\11.2.0.4\dbhome_1\bin\oraclr11.dll")

                )

                (SID_DESC =

                  (GLOBAL_DBNAME = dbstater)

                  (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)

                  (SID_NAME = dbstater)

                )

                (SID_DESC =

                  (GLOBAL_DBNAME = dbstater_DGMGRL)

                  (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)

                  (SID_NAME = dbstater)

                )

              )

             

            LISTENER =

              (DESCRIPTION_LIST =

                (DESCRIPTION =

                  (ADDRESS = (PROTOCOL = TCP)(HOST = SRV7-W2012)(PORT = 1521))

                  (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))

                )

              )


            ADR_BASE_LISTENER = C:\app\oracle

            INBOUND_CONNECT_TIMEOUT_LISTENER=120

             

            listener.ora on standby server:

            # listener.ora Network Configuration File: C:\app\oracle\product\11.2.0.4\dbhome_1\network\admin\listener.ora

            # Generated by Oracle configuration tools.

             

             

            SID_LIST_LISTENER =

              (SID_LIST =

                (SID_DESC =

                  (SID_NAME = CLRExtProc)

                  (ORACLE_HOME = C:\app\oracle\product\11.2.0.4\dbhome_1)

                  (PROGRAM = extproc)

                  (ENVS = "EXTPROC_DLLS=ONLY:C:\app\oracle\product\11.2.0.4\dbhome_1\bin\oraclr11.dll")

                )

                (SID_DESC =

                  (GLOBAL_DBNAME = dbstatst)

                  (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)

                  (SID_NAME = dbstatst)

                )

                (SID_DESC =

                  (GLOBAL_DBNAME = dbstatst_DGMGRL)

                  (ORACLE_HOME = /oracle/product/11.2.0/dbhome_1)

                  (SID_NAME = dbstatst)

                )

              )

             

            LISTENER =

              (DESCRIPTION_LIST =

                (DESCRIPTION =

                  (ADDRESS = (PROTOCOL = TCP)(HOST = SRV10-W2012)(PORT = 1521))

                  (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))

                )

              )

             

            ADR_BASE_LISTENER = C:\app\oracle

            INBOUND_CONNECT_TIMEOUT_LISTENER=120

             

            tnsnames.ora

            # tnsnames.ora Network Configuration File: C:\app\oracle\product\11.2.0.4\dbhome_1\network\admin\tnsnames.ora

            # Generated by Oracle configuration tools.

             

             

            DBSTATER =

              (DESCRIPTION =

                (ADDRESS = (PROTOCOL = TCP)(HOST = SRV7-W2012)(PORT = 1521))

                (CONNECT_DATA =

                  (SERVER = DEDICATED)

                  (SERVICE_NAME = DBSTATER)

                )

              )

             

            ORACLR_CONNECTION_DATA =

              (DESCRIPTION =

                (ADDRESS_LIST =

                  (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))

                )

                (CONNECT_DATA =

                  (SID = CLRExtProc)

                  (PRESENTATION = RO)

                )

              )

             

            DBSTATST =

              (DESCRIPTION =

                (ADDRESS = (PROTOCOL = TCP)(HOST = SRV10-W2012)(PORT = 1521))

                (CONNECT_DATA =

                  (SERVER = DEDICATED)

                  (SERVICE_NAME = dbstatst)

                )

              )

             

            So we can see errors in the standby site but, at the end, the roles are OK and the synchronisation is also OK.

             

            I tested that configuration in Win 2088 and the same problem appeared.

             

            If I switch back, the same errors appear on the primary server which is the standby database.

            One precision : these tests were executed on 2 VMs (Virtualbox) and the name resolution is performed by hosts file.

             

            Thanks for your help

             

            Christophe

            • 3. Re: Errors during Databaguard broker switchover
              HristiyanK

              Hello Christophe,

               

              You have created static entries in listener.ora with *_DGMGRL but I can't see them in tnsnames.ora. Can you share how you've created DGMGRL configuration DBSTATER_DBSTATST. What connect identifier you used?

              One more thing crossed my mind about listener.ora. I think you may try to add "TNS_ADMIN=<db home>/network/admin" into the ENVS part.

              Name resolution should be fine with hosts file.

              Sorry for the lack of explanations, but I don't have much time right now as its the beginning of a weekend Let me know if you have some doubts.

              Looking forward to your reply!

               

              Regards,

              Hristiyan