This discussion is archived
11 Replies Latest reply: Jan 4, 2013 1:25 AM by Ian Hoogeboom RSS

Changing host Time backwards - two node RAC

968673 Newbie
Currently Being Moderated
Hello, We have a two node RAC enviornment. Oracle Version: 11.2.0.1, Operating System Microsoft Windows x64 (64-bit) 2003 Server R2

On one of the nodes, host time is ahead by 30 minutues.

What is the impact on the RAC enviornment on setting the clock backwards?

I have read that one should not set the clock backwards, but not sure why.

Cluster Time Synchronization Service (CTSS) is in observer mode. No errors when I run cluvf

Verifying Clock Synchronization across the cluster nodes

Checking if Clusterware is installed on all nodes...
Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...
Check: CTSS Resource running on all nodes

Node Name Status
------------------------------------ ------------------------
xx-11rac2 passed
xx-11rac1 passed
Result: CTSS resource check passed


Querying CTSS for time offset on all nodes...
Result: Query of CTSS for time offset passed

Check CTSS state started...
Check: CTSS state

Node Name State
------------------------------------ ------------------------
xx-11rac2 Observer
xx-11rac1 Observer
CTSS is in Observer state. Switching over to clock synchronization checks using NTP


Checking daemon liveness...

Check: Liveness for "W32Time"
Node Name Running?
------------------------------------ ------------------------
xx-11rac2 yes
xx-11rac1 yes
Result: Liveness check passed for "W32Time"

Oracle Cluster Time Synchronization Services check passed

Verification of Clock Synchronization across the cluster nodes was successful.
  • 1. Re: Changing host Time backwards - two node RAC
    Sebastian Solbach (DBA Community) Guru
    Currently Being Moderated
    Hi,

    if you set the clock backwards while clusterware is running, the monitoring processes of clusterware processes fail, since they "think" they detected a scheduling problem of the root processes.
    This will lead to a node reboot.

    Hence stop clusterware before changing the clock.

    Regards
    Sebastian
  • 2. Re: Changing host Time backwards - two node RAC
    968673 Newbie
    Currently Being Moderated
    Sebastian,

    Thanks very much for your reply, good useful information.

    Other than the node reboot, should I worry about any timestamp issue, any data related issues?

    Reading the RAC documentation and other blogs, they all make a big deal about making sure Time is NOT adjusted backwards.

    What are some of the issues with time being adjusted backwards, wouldn't the node eviction/reboot also happen with time being adjusted forward?


    Here is a blog entry:

    Synchronizing the Time on ALL Nodes
    There is a general requirement for Oracle RAC that the time on all the nodes be the same. With 11gR2 time
    synchronization can be performed by the Clusterware using CTSSD (Cluster Time Synchronization Services
    Daemon) or by using the Windows Time Service. If the Windows Time Service is being used, it MUST be
    configured to prevent the time from being adjusted backwards. Perform the following steps to ensure the time
    is NOT adjusted backwards using Windows Time Service:

    Open a command prompt and type "regedit"
    Within the registry editor locate the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config key.
    Set the value for MaxNegPhaseCorrection? to 0 and exit the registry editor.
    Open a command prompt and execute the following to put the change into effect:
    cmd> W32tm /config /update

    Repeat steps 1 through 4 for ALL cluster nodes.
  • 3. Re: Changing host Time backwards - two node RAC
    Sebastian Solbach (DBA Community) Guru
    Currently Being Moderated
    Hi,

    this is for automatic time synchronisation during runtime.
    If you setup NTP, NTP may also set the time backwards - and since it is an automatic process it may happen more often.
    Hence you must take care that no automatic process switches the time backward while clusterware is running.

    If you do it on administration prurpose - shut down clusterware and the set the time backwards, then the only issue you may face is a time "jump" in the logfiles, making it more difficult to analyze them ;)

    Regards

    Sebastian
  • 4. Re: Changing host Time backwards - two node RAC
    Levi-Pereira Guru
    Currently Being Moderated
    On one of the nodes, host time is ahead by 30 minutues.
       Node Name                             State                   
       ------------------------------------  ------------------------
       xx-11rac2                         Observer                
       xx-11rac1                         Observer                
     CTSS is in Observer state. Switching over to clock synchronization checks using NTP
     
     
     Checking daemon liveness...
     
     Check: Liveness for "W32Time"
       Node Name                             Running?                
       ------------------------------------  ------------------------
       xx-11rac2                         yes                     
       xx-11rac1                         yes                     
     Result: Liveness check passed for "W32Time"
    How do you have a service time synchronization "W32Time" active and your cluster have 30 minutes difference from one node to the other?

    This service "W32Time" should on minimum prevent it happening.

    P.S: The timezone must be configured the same on all nodes and daylight saving time adjustments do not affect the system clock

    This reply was only a comment in your configuration.
    (Sebastian already answered your question)
  • 5. Re: Changing host Time backwards - two node RAC
    968673 Newbie
    Currently Being Moderated
    Sebastian,

    Thanks again for your detailed reply.

    Much better explanation than the one I received from Oracle support, our friend at Oracle support merely directed me to a RAC FAQ -:) The correct answere nowhere to be found.

    So I came up with the steps to do this

    1) shutdown clusterware

    2) stop/start the w32time service [H:\>net stop w32time , H:\>net start w32time]

    3) resync time with Network domain controller [H:\>w32tm/resync ]

    4) startup clusterware
  • 6. Re: Changing host Time backwards - two node RAC
    968673 Newbie
    Currently Being Moderated
    Levi,

    Your question prompted some research. At first I thought it was just a 'resync' problem with domain control server.

    Looking into the registery it appears, a setting is changed to prevent father time from going backwards

    value for MaxNegPhaseCorrection is set to 0(zero)

    So I think this explains why one of the nodes is not able to re-sync

    Notes;
    Open a command prompt and type "regedit"
    Within the registry editor locate the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config key.
    Set the value for MaxNegPhaseCorrection? to 0 and exit the registry editor.
    Open a command prompt and execute the following to put the change into effect:
    cmd> W32tm /config /update
  • 7. Re: Changing host Time backwards - two node RAC
    Levi-Pereira Guru
    Currently Being Moderated
    I've performed this setting before and Oracle recommends something slightly different.

    *RAC and Oracle Clusterware Best Practices and Starter Kit (Windows) [ID 811271.1]*
    value for MaxPosPhaseCorrection = 600
    value for MaxNegPhaseCorrection = 600
    value for MaxAllowedPhaseOffset = 600
  • 8. Re: Changing host Time backwards - two node RAC
    698658 Newbie
    Currently Being Moderated
    Sebastian,
    what about time zone changes when You're going 1hour back here in Poland :).
    Does the cluster reboot ?
    Regards
    GregG
  • 9. Re: Changing host Time backwards - two node RAC
    Sebastian Solbach (DBA Community) Guru
    Currently Being Moderated
    Hi GregG,

    time zone changes do not change the time of the internal clock of your computer (UTC). It just changes the time it displays. Hence not a problem.

    However the "leap second" on Years end (every some years or so) poses this problem:

    NTP leap second event causing Oracle Clusterware node reboot     [Document 759143.1]     

    Regards
    Sebastian
  • 10. Re: Changing host Time backwards - two node RAC
    698658 Newbie
    Currently Being Moderated
    Thank You, that make a lot of sense :).
    Regards
    GregG
  • 11. Re: Changing host Time backwards - two node RAC
    Ian Hoogeboom Newbie
    Currently Being Moderated
    Levi Pereira wrote:
    I've performed this setting before and Oracle recommends something slightly different.

    *RAC and Oracle Clusterware Best Practices and Starter Kit (Windows) [ID 811271.1]*
    value for MaxPosPhaseCorrection = 600
    value for MaxNegPhaseCorrection = 600
    value for MaxAllowedPhaseOffset = 600
    That is different from the [Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Microsoft Windows Part Number E10817-01]

    http://docs.oracle.com/cd/E14848_01/doc/install.112/e10817/prewin.htm#BABJBHFH
    Locate the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config key.
    Set the value for MaxNegPhaseCorrection to 0.
    I thought hardware are never allowed to go back in time (for there cluster cases). If they need to correct this, they should run the clock faster.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points