4 Replies Latest reply: Feb 19, 2013 6:45 AM by user2638923 RSS

    Switchover succeeds but observer stops. Only happens one way.

    user2638923
      Hello

      I have a problem with the following setup:

      Oracle Enterprise Edition, version - 11.2.0.3
      OS - RHEL6 64bit

      Configuration:

      2x hosts.
      2x databases on each host.
      each database forms 1/2 of a PRIMARY + PHYSICAL STANDBY 'pair'.
      Mode = Maximum Availability.
      FSFO is setup using a third host for the observer location.
      Grid Control 11g


      This setup is working fine on our acceptance test hosts (same software, spec etc.) and is also working fine for one of the two db pairs on these machines. However, for one of the pairs, a switchover one way will stop the observer and it does not restart. i.e.

      we have host A and host B which have database 'pairs' for
      SOA
      In House db

      Primary db switchover to result
      soadba soadbb success
      soadbb soadba success
      inhousea inhouseb switchover succeeds, inhousea is reisnstated as physical standby but observer stops
      inhouseb inhousea success


      The contents of the observer log is:

      Observer started
      [W000 02/18 14:10:26.32] Observer started.
      [P003 02/18 14:13:25.83] Authentication failed.
      Observer stopped


      This behaviour is consistent whichever way I invoke the switchover e.g.
      DGMGRL on host A
      DGMGRL on host B
      Grid Control

      I can sqlplus from everywhere to everywhere that I can think of using the range of connect identifiers I would expect:
      @inhousea as sysdba
      @inhousea_dgmgrl as sysdba
      @inhousea_dgmgrl.acme.com as sysdba

      Credentials are stored in Grid Control for hosts and dbs and all test okay. Static registration with default Grid listener seems to be working.



      Any ideas where I should look next? I don't believe it is a typo, and in fact have just copied the 2xlistener (GRID and DB HOME) and 1xtnsnames.ora file from host A to host B so is there some other area of connectivity I'm not accounting for. Observer location on the third host is used by both the inhouse and the soa configurations so the problem is not with the ORACLE_HOME in use there (or at least the ORACLE_HOME on the third host can do the job).


      Any help much appreciated as I'm scratching my head over this.



      thanks
      Mark
        • 1. Re: Switchover succeeds but observer stops. Only happens one way.
          CKPT
          What are the errors you have faced?
          You able to fetch much information from Broker log files?
          When you start observer, What the information from observer log files?
          hope flashback is enabled on both primary and standby databases.

          Can you share
          DGMGRL> show configuration verbose;
          DGMGRL> show database verbose <prim>;
          DGMGRL> show database verbose <standby>;
          • 2. Re: Switchover succeeds but observer stops. Only happens one way.
            user2638923
            Hello CKPT


            Thanks for the quick response.


            The error is as below:

            DGMGRL> show configuration

            Configuration - inhousea

            Protection Mode: MaxAvailability
            Databases:
            inhouseb - Primary database
            Error: ORA-16820: fast-start failover observer is no longer observing this database

            inhousea - (*) Physical standby database
            Error: ORA-16820: fast-start failover observer is no longer observing this database

            Fast-Start Failover: ENABLED

            Configuration Status:
            ERROR

            Following that I can restart the observer through OEM and it stays up. But that's not really the point, it should survive a switchover and in fact does if I'm going from PRIMARY=inhouseb to PRIMARY=inhousea


            select flashback_on from v$database ;

            FLASHBACK_ON
            ------------------
            YES

            (same on both dbs)


            drc*.log contents for the end of the switchover are:

            HOST A
            Starting Data Guard Broker bootstrap <<
            Broker Configuration File Locations:
            dg_broker_config_file1 = "/opt/oracle_homes/11.2.0.3.0/dbhome_1/dbs/dr1inhousea.dat"
            dg_broker_config_file2 = "/fs2/control/inhousea/dr2inhousea.dat"
            02/18/2013 16:33:38
            DMON Registering service inhousea_DGB with listener(s)
            Broker Configuration: "inhousea"
            Protection Mode: Maximum Availability
            Fast-Start Failover (FSFO): Enabled, flags=0x41001, version=240
            Primary Database: inhouseb (0x02010000)
            Standby Database: inhousea, Enabled Physical Standby (FSFO target) (0x01010000)
            02/18/2013 16:33:42
            Creating process RSM0
            02/18/2013 16:33:58
            Notifying Oracle Clusterware to buildup standby database after SWITCHOVER
            Data Guard notifying Oracle Clusterware to start services and other instances change
            Command SWITCHOVER TO inhousea completed




            HOST B

            Notifying Oracle Clusterware to teardown target standby database for SWITCHOVER
            02/18/2013 16:33:24
            Command SWITCHOVER TO inhouseb completed
            02/18/2013 16:33:27
            Failed to connect to remote database inhousea. Error is ORA-12514
            Failed to connect to remote database inhousea. Error is ORA-01034
            02/18/2013 16:33:28
            LGWR: FSFO SetState("UNSYNC", 0x2) operation requires an ack
            LGWR: Supplying requested ack on behalf of old primary that is in the progress of the switchover.
            LGWR: FSFO ack received
            02/18/2013 16:33:31

            Deferring associated archivelog destinations of sites permanently disabled due to Switchover
            Notifying Oracle Clusterware to buildup primary database after SWITCHOVER
            Data Guard notifying Oracle Clusterware to start services and other instances change
            Command SWITCHOVER TO inhouseb completed
            Redo transport problem detected: redo transport for database inhousea has the following error:
            ORA-01034: ORACLE not available
            02/18/2013 16:33:33
            Failed to connect to remote database inhousea. Error is ORA-01034
            Failed to send message to site inhousea. Error code is ORA-01034.
            Data Guard Broker Status Summary:
            Type Name Severity Status
            Configuration inhousea Warning ORA-16607
            Primary Database inhouseb Error ORA-16825
            Physical Standby Database inhousea Error ORA-01034
            02/18/2013 16:33:58
            Deferring associated archivelog destinations of sites permanently disabled due to Switchover
            Notifying Oracle Clusterware to buildup primary database after SWITCHOVER
            Data Guard notifying Oracle Clusterware to start services and other instances change
            Command SWITCHOVER TO inhousea completed
            02/18/2013 16:34:29
            Data Guard Broker Status Summary:
            Type Name Severity Status
            Configuration inhousea Warning ORA-16607
            Primary Database inhouseb Error ORA-16820
            Physical Standby Database inhousea Error ORA-16820
            02/18/2013 16:35:17
            Data Guard Broker Status Summary:
            Type Name Severity Status
            Configuration inhousea Warning ORA-16607
            Primary Database inhouseb Error ORA-16820
            Physical Standby Database inhousea Error ORA-16820
            02/18/2013 16:36:17
            Data Guard Broker Status Summary:
            Type Name Severity Status
            Configuration inhousea Warning ORA-16607
            Primary Database inhouseb Error ORA-16820
            Physical Standby Database inhousea Error ORA-16820
            02/18/2013 16:37:04



            the observer went down at 16:33:26:
            Observer started
            [W000 02/18 14:24:08.56] Observer started.
            [P003 02/18 16:33:26.34] Authentication failed.
            Observer stopped


            I would expect (I think) the odd ORA-12514 etc while the old PRIMARY is being restarted but not for the observer to die like it does. Then after that it's moaning about the lack of observer (and rightly so!).



            thanks
            Mark
            • 3. Re: Switchover succeeds but observer stops. Only happens one way.
              CKPT
              Failed to connect to remote database inhousea. Error is ORA-12514
              So the services aren't registered with listener. Have you configured static listener?
              and you should able to see in $lsnrctl services/ $lsnrctl status. If you able to fix this then broker should able to communicate all the databases in the configuraiton.
              If possible share your both listener and tnsnames.ora files
              12514, 00000, "TNS:listener does not currently know of service requested in connect descriptor"
              // *Cause:  The listener received a request to establish a connection to a
              // database or other service. The connect descriptor received by the listener
              // specified a service name for a service (usually a database service)
              // that either has not yet dynamically registered with the listener or has
              // not been statically configured for the listener.  This may be a temporary
              // condition such as after the listener has started, but before the database
              // instance has registered with the listener.
              // *Action:
              //  - Wait a moment and try to connect a second time.
              //  - Check which services are currently known by the listener by executing:
              //    lsnrctl services <listener name>
              //  - Check that the SERVICE_NAME parameter in the connect descriptor of the
              //    net service name used specifies a service known by the listener.
              //  - If an easy connect naming connect identifier was used, check that
              //    the service name specified is a service known by the listener.
              //  - Check for an event in the listener.log file.
              • 4. Re: Switchover succeeds but observer stops. Only happens one way.
                user2638923
                CKPT

                Thank you! I was convinced everything was properly registered with the listener but it turned out that one out of the 6 was not. I have redone all the listener.ora files and it now seems to be working.

                Thanks for taking the time to look at this, a second opinion confirming it was in the area of authentication put me on the right track.

                From now on I will pay closer attention to those ORA-12514 messages!


                Mark

                Edited by: user2638923 on 19-Feb-2013 04:44