11 Replies Latest reply: Jan 4, 2013 3:25 AM by Ian Hoogeboom RSS

    Changing host Time backwards - two node RAC

    968673
      Hello, We have a two node RAC enviornment. Oracle Version: 11.2.0.1, Operating System Microsoft Windows x64 (64-bit) 2003 Server R2

      On one of the nodes, host time is ahead by 30 minutues.

      What is the impact on the RAC enviornment on setting the clock backwards?

      I have read that one should not set the clock backwards, but not sure why.

      Cluster Time Synchronization Service (CTSS) is in observer mode. No errors when I run cluvf

      Verifying Clock Synchronization across the cluster nodes

      Checking if Clusterware is installed on all nodes...
      Check of Clusterware install passed

      Checking if CTSS Resource is running on all nodes...
      Check: CTSS Resource running on all nodes

      Node Name Status
      ------------------------------------ ------------------------
      xx-11rac2 passed
      xx-11rac1 passed
      Result: CTSS resource check passed


      Querying CTSS for time offset on all nodes...
      Result: Query of CTSS for time offset passed

      Check CTSS state started...
      Check: CTSS state

      Node Name State
      ------------------------------------ ------------------------
      xx-11rac2 Observer
      xx-11rac1 Observer
      CTSS is in Observer state. Switching over to clock synchronization checks using NTP


      Checking daemon liveness...

      Check: Liveness for "W32Time"
      Node Name Running?
      ------------------------------------ ------------------------
      xx-11rac2 yes
      xx-11rac1 yes
      Result: Liveness check passed for "W32Time"

      Oracle Cluster Time Synchronization Services check passed

      Verification of Clock Synchronization across the cluster nodes was successful.
        • 1. Re: Changing host Time backwards - two node RAC
          Sebastian Solbach -Dba Community-Oracle
          Hi,

          if you set the clock backwards while clusterware is running, the monitoring processes of clusterware processes fail, since they "think" they detected a scheduling problem of the root processes.
          This will lead to a node reboot.

          Hence stop clusterware before changing the clock.

          Regards
          Sebastian
          • 2. Re: Changing host Time backwards - two node RAC
            968673
            Sebastian,

            Thanks very much for your reply, good useful information.

            Other than the node reboot, should I worry about any timestamp issue, any data related issues?

            Reading the RAC documentation and other blogs, they all make a big deal about making sure Time is NOT adjusted backwards.

            What are some of the issues with time being adjusted backwards, wouldn't the node eviction/reboot also happen with time being adjusted forward?


            Here is a blog entry:

            Synchronizing the Time on ALL Nodes
            There is a general requirement for Oracle RAC that the time on all the nodes be the same. With 11gR2 time
            synchronization can be performed by the Clusterware using CTSSD (Cluster Time Synchronization Services
            Daemon) or by using the Windows Time Service. If the Windows Time Service is being used, it MUST be
            configured to prevent the time from being adjusted backwards. Perform the following steps to ensure the time
            is NOT adjusted backwards using Windows Time Service:

            Open a command prompt and type "regedit"
            Within the registry editor locate the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config key.
            Set the value for MaxNegPhaseCorrection? to 0 and exit the registry editor.
            Open a command prompt and execute the following to put the change into effect:
            cmd> W32tm /config /update

            Repeat steps 1 through 4 for ALL cluster nodes.
            • 3. Re: Changing host Time backwards - two node RAC
              Sebastian Solbach -Dba Community-Oracle
              Hi,

              this is for automatic time synchronisation during runtime.
              If you setup NTP, NTP may also set the time backwards - and since it is an automatic process it may happen more often.
              Hence you must take care that no automatic process switches the time backward while clusterware is running.

              If you do it on administration prurpose - shut down clusterware and the set the time backwards, then the only issue you may face is a time "jump" in the logfiles, making it more difficult to analyze them ;)

              Regards

              Sebastian
              • 4. Re: Changing host Time backwards - two node RAC
                Levi Pereira
                On one of the nodes, host time is ahead by 30 minutues.
                   Node Name                             State                   
                   ------------------------------------  ------------------------
                   xx-11rac2                         Observer                
                   xx-11rac1                         Observer                
                 CTSS is in Observer state. Switching over to clock synchronization checks using NTP
                 
                 
                 Checking daemon liveness...
                 
                 Check: Liveness for "W32Time"
                   Node Name                             Running?                
                   ------------------------------------  ------------------------
                   xx-11rac2                         yes                     
                   xx-11rac1                         yes                     
                 Result: Liveness check passed for "W32Time"
                How do you have a service time synchronization "W32Time" active and your cluster have 30 minutes difference from one node to the other?

                This service "W32Time" should on minimum prevent it happening.

                P.S: The timezone must be configured the same on all nodes and daylight saving time adjustments do not affect the system clock

                This reply was only a comment in your configuration.
                (Sebastian already answered your question)
                • 5. Re: Changing host Time backwards - two node RAC
                  968673
                  Sebastian,

                  Thanks again for your detailed reply.

                  Much better explanation than the one I received from Oracle support, our friend at Oracle support merely directed me to a RAC FAQ -:) The correct answere nowhere to be found.

                  So I came up with the steps to do this

                  1) shutdown clusterware

                  2) stop/start the w32time service [H:\>net stop w32time , H:\>net start w32time]

                  3) resync time with Network domain controller [H:\>w32tm/resync ]

                  4) startup clusterware
                  • 6. Re: Changing host Time backwards - two node RAC
                    968673
                    Levi,

                    Your question prompted some research. At first I thought it was just a 'resync' problem with domain control server.

                    Looking into the registery it appears, a setting is changed to prevent father time from going backwards

                    value for MaxNegPhaseCorrection is set to 0(zero)

                    So I think this explains why one of the nodes is not able to re-sync

                    Notes;
                    Open a command prompt and type "regedit"
                    Within the registry editor locate the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config key.
                    Set the value for MaxNegPhaseCorrection? to 0 and exit the registry editor.
                    Open a command prompt and execute the following to put the change into effect:
                    cmd> W32tm /config /update
                    • 7. Re: Changing host Time backwards - two node RAC
                      Levi Pereira
                      I've performed this setting before and Oracle recommends something slightly different.

                      *RAC and Oracle Clusterware Best Practices and Starter Kit (Windows) [ID 811271.1]*
                      value for MaxPosPhaseCorrection = 600
                      value for MaxNegPhaseCorrection = 600
                      value for MaxAllowedPhaseOffset = 600
                      • 8. Re: Changing host Time backwards - two node RAC
                        698658
                        Sebastian,
                        what about time zone changes when You're going 1hour back here in Poland :).
                        Does the cluster reboot ?
                        Regards
                        GregG
                        • 9. Re: Changing host Time backwards - two node RAC
                          Sebastian Solbach -Dba Community-Oracle
                          Hi GregG,

                          time zone changes do not change the time of the internal clock of your computer (UTC). It just changes the time it displays. Hence not a problem.

                          However the "leap second" on Years end (every some years or so) poses this problem:

                          NTP leap second event causing Oracle Clusterware node reboot     [Document 759143.1]     

                          Regards
                          Sebastian
                          • 10. Re: Changing host Time backwards - two node RAC
                            698658
                            Thank You, that make a lot of sense :).
                            Regards
                            GregG
                            • 11. Re: Changing host Time backwards - two node RAC
                              Ian Hoogeboom
                              Levi Pereira wrote:
                              I've performed this setting before and Oracle recommends something slightly different.

                              *RAC and Oracle Clusterware Best Practices and Starter Kit (Windows) [ID 811271.1]*
                              value for MaxPosPhaseCorrection = 600
                              value for MaxNegPhaseCorrection = 600
                              value for MaxAllowedPhaseOffset = 600
                              That is different from the [Oracle® Grid Infrastructure Installation Guide 11g Release 2 (11.2) for Microsoft Windows Part Number E10817-01]

                              http://docs.oracle.com/cd/E14848_01/doc/install.112/e10817/prewin.htm#BABJBHFH
                              Locate the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\W32Time\Config key.
                              Set the value for MaxNegPhaseCorrection to 0.
                              I thought hardware are never allowed to go back in time (for there cluster cases). If they need to correct this, they should run the clock faster.