This discussion is archived
4 Replies Latest reply: Feb 19, 2013 4:45 AM by user2638923 RSS

Switchover succeeds but observer stops. Only happens one way.

user2638923 Newbie
Currently Being Moderated
Hello

I have a problem with the following setup:

Oracle Enterprise Edition, version - 11.2.0.3
OS - RHEL6 64bit

Configuration:

2x hosts.
2x databases on each host.
each database forms 1/2 of a PRIMARY + PHYSICAL STANDBY 'pair'.
Mode = Maximum Availability.
FSFO is setup using a third host for the observer location.
Grid Control 11g


This setup is working fine on our acceptance test hosts (same software, spec etc.) and is also working fine for one of the two db pairs on these machines. However, for one of the pairs, a switchover one way will stop the observer and it does not restart. i.e.

we have host A and host B which have database 'pairs' for
SOA
In House db

Primary db switchover to result
soadba soadbb success
soadbb soadba success
inhousea inhouseb switchover succeeds, inhousea is reisnstated as physical standby but observer stops
inhouseb inhousea success


The contents of the observer log is:

Observer started
[W000 02/18 14:10:26.32] Observer started.
[P003 02/18 14:13:25.83] Authentication failed.
Observer stopped


This behaviour is consistent whichever way I invoke the switchover e.g.
DGMGRL on host A
DGMGRL on host B
Grid Control

I can sqlplus from everywhere to everywhere that I can think of using the range of connect identifiers I would expect:
@inhousea as sysdba
@inhousea_dgmgrl as sysdba
@inhousea_dgmgrl.acme.com as sysdba

Credentials are stored in Grid Control for hosts and dbs and all test okay. Static registration with default Grid listener seems to be working.



Any ideas where I should look next? I don't believe it is a typo, and in fact have just copied the 2xlistener (GRID and DB HOME) and 1xtnsnames.ora file from host A to host B so is there some other area of connectivity I'm not accounting for. Observer location on the third host is used by both the inhouse and the soa configurations so the problem is not with the ORACLE_HOME in use there (or at least the ORACLE_HOME on the third host can do the job).


Any help much appreciated as I'm scratching my head over this.



thanks
Mark
  • 1. Re: Switchover succeeds but observer stops. Only happens one way.
    CKPT Guru
    Currently Being Moderated
    What are the errors you have faced?
    You able to fetch much information from Broker log files?
    When you start observer, What the information from observer log files?
    hope flashback is enabled on both primary and standby databases.

    Can you share
    DGMGRL> show configuration verbose;
    DGMGRL> show database verbose <prim>;
    DGMGRL> show database verbose <standby>;
  • 2. Re: Switchover succeeds but observer stops. Only happens one way.
    user2638923 Newbie
    Currently Being Moderated
    Hello CKPT


    Thanks for the quick response.


    The error is as below:

    DGMGRL> show configuration

    Configuration - inhousea

    Protection Mode: MaxAvailability
    Databases:
    inhouseb - Primary database
    Error: ORA-16820: fast-start failover observer is no longer observing this database

    inhousea - (*) Physical standby database
    Error: ORA-16820: fast-start failover observer is no longer observing this database

    Fast-Start Failover: ENABLED

    Configuration Status:
    ERROR

    Following that I can restart the observer through OEM and it stays up. But that's not really the point, it should survive a switchover and in fact does if I'm going from PRIMARY=inhouseb to PRIMARY=inhousea


    select flashback_on from v$database ;

    FLASHBACK_ON
    ------------------
    YES

    (same on both dbs)


    drc*.log contents for the end of the switchover are:

    HOST A
    Starting Data Guard Broker bootstrap <<
    Broker Configuration File Locations:
    dg_broker_config_file1 = "/opt/oracle_homes/11.2.0.3.0/dbhome_1/dbs/dr1inhousea.dat"
    dg_broker_config_file2 = "/fs2/control/inhousea/dr2inhousea.dat"
    02/18/2013 16:33:38
    DMON Registering service inhousea_DGB with listener(s)
    Broker Configuration: "inhousea"
    Protection Mode: Maximum Availability
    Fast-Start Failover (FSFO): Enabled, flags=0x41001, version=240
    Primary Database: inhouseb (0x02010000)
    Standby Database: inhousea, Enabled Physical Standby (FSFO target) (0x01010000)
    02/18/2013 16:33:42
    Creating process RSM0
    02/18/2013 16:33:58
    Notifying Oracle Clusterware to buildup standby database after SWITCHOVER
    Data Guard notifying Oracle Clusterware to start services and other instances change
    Command SWITCHOVER TO inhousea completed




    HOST B

    Notifying Oracle Clusterware to teardown target standby database for SWITCHOVER
    02/18/2013 16:33:24
    Command SWITCHOVER TO inhouseb completed
    02/18/2013 16:33:27
    Failed to connect to remote database inhousea. Error is ORA-12514
    Failed to connect to remote database inhousea. Error is ORA-01034
    02/18/2013 16:33:28
    LGWR: FSFO SetState("UNSYNC", 0x2) operation requires an ack
    LGWR: Supplying requested ack on behalf of old primary that is in the progress of the switchover.
    LGWR: FSFO ack received
    02/18/2013 16:33:31

    Deferring associated archivelog destinations of sites permanently disabled due to Switchover
    Notifying Oracle Clusterware to buildup primary database after SWITCHOVER
    Data Guard notifying Oracle Clusterware to start services and other instances change
    Command SWITCHOVER TO inhouseb completed
    Redo transport problem detected: redo transport for database inhousea has the following error:
    ORA-01034: ORACLE not available
    02/18/2013 16:33:33
    Failed to connect to remote database inhousea. Error is ORA-01034
    Failed to send message to site inhousea. Error code is ORA-01034.
    Data Guard Broker Status Summary:
    Type Name Severity Status
    Configuration inhousea Warning ORA-16607
    Primary Database inhouseb Error ORA-16825
    Physical Standby Database inhousea Error ORA-01034
    02/18/2013 16:33:58
    Deferring associated archivelog destinations of sites permanently disabled due to Switchover
    Notifying Oracle Clusterware to buildup primary database after SWITCHOVER
    Data Guard notifying Oracle Clusterware to start services and other instances change
    Command SWITCHOVER TO inhousea completed
    02/18/2013 16:34:29
    Data Guard Broker Status Summary:
    Type Name Severity Status
    Configuration inhousea Warning ORA-16607
    Primary Database inhouseb Error ORA-16820
    Physical Standby Database inhousea Error ORA-16820
    02/18/2013 16:35:17
    Data Guard Broker Status Summary:
    Type Name Severity Status
    Configuration inhousea Warning ORA-16607
    Primary Database inhouseb Error ORA-16820
    Physical Standby Database inhousea Error ORA-16820
    02/18/2013 16:36:17
    Data Guard Broker Status Summary:
    Type Name Severity Status
    Configuration inhousea Warning ORA-16607
    Primary Database inhouseb Error ORA-16820
    Physical Standby Database inhousea Error ORA-16820
    02/18/2013 16:37:04



    the observer went down at 16:33:26:
    Observer started
    [W000 02/18 14:24:08.56] Observer started.
    [P003 02/18 16:33:26.34] Authentication failed.
    Observer stopped


    I would expect (I think) the odd ORA-12514 etc while the old PRIMARY is being restarted but not for the observer to die like it does. Then after that it's moaning about the lack of observer (and rightly so!).



    thanks
    Mark
  • 3. Re: Switchover succeeds but observer stops. Only happens one way.
    CKPT Guru
    Currently Being Moderated
    Failed to connect to remote database inhousea. Error is ORA-12514
    So the services aren't registered with listener. Have you configured static listener?
    and you should able to see in $lsnrctl services/ $lsnrctl status. If you able to fix this then broker should able to communicate all the databases in the configuraiton.
    If possible share your both listener and tnsnames.ora files
    12514, 00000, "TNS:listener does not currently know of service requested in connect descriptor"
    // *Cause:  The listener received a request to establish a connection to a
    // database or other service. The connect descriptor received by the listener
    // specified a service name for a service (usually a database service)
    // that either has not yet dynamically registered with the listener or has
    // not been statically configured for the listener.  This may be a temporary
    // condition such as after the listener has started, but before the database
    // instance has registered with the listener.
    // *Action:
    //  - Wait a moment and try to connect a second time.
    //  - Check which services are currently known by the listener by executing:
    //    lsnrctl services <listener name>
    //  - Check that the SERVICE_NAME parameter in the connect descriptor of the
    //    net service name used specifies a service known by the listener.
    //  - If an easy connect naming connect identifier was used, check that
    //    the service name specified is a service known by the listener.
    //  - Check for an event in the listener.log file.
  • 4. Re: Switchover succeeds but observer stops. Only happens one way.
    user2638923 Newbie
    Currently Being Moderated
    CKPT

    Thank you! I was convinced everything was properly registered with the listener but it turned out that one out of the 6 was not. I have redone all the listener.ora files and it now seems to be working.

    Thanks for taking the time to look at this, a second opinion confirming it was in the area of authentication put me on the right track.

    From now on I will pay closer attention to those ORA-12514 messages!


    Mark

    Edited by: user2638923 on 19-Feb-2013 04:44

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points