12 Replies Latest reply on Oct 17, 2012 9:47 PM by J Peters-Oracle

    OEL6 System Always Boots w/ Bad Time

    Rob
      Good Afternoon,

      I have an OEL6 box sitting in OVM (3.1.1) that is consistently booting with a clock that is several hours off... as though it thinks the hardware clock is set to UTC while it's set to local time.

      Since VMs don't have hardware clocks, I'm not sure how to fix this. I've tried running system-config-date and un-checking the UTC checkbox to no avail. I've also tried editing /etc/sysconfig/clock, setting UTC=FALSE and UTC=TRUE, but nothing seems to have any impact. The only "workaround" I have thus far is to manually run ntpdate every time it boots, as it's so far off even ntpd won't fix it.

      I'm stumped... any ideas?

      Rob

      Edited by: Rob on Jul 31, 2012 3:23 PM
        • 1. Re: OEL6 System Always Boots w/ Bad Time
          929328
          It is a common problem.
          Caution! Wall of text!
          E.7 Wallclock Time Skew Problems

          Oracle VM Release 2.1.1 introduces the use of the timer_mode parameter for hardware virtualized guests. This parameter, when properly applied, can reduce or even eliminate problems with wallclock time skew in most hardware virtualized guests. Wallclock time skew problems do not occur in paravirtualized guests.

          Since the application of the correct value of the timer_mode parameter can be difficult to determine, you can pass the os-type and os-variant command-line switches to virt-install to select the best timer_mode value for the guest operating system. When you use these virt-install parameters, the correct timer_mode value is automatically added to the guest configuration file. For example, to create an Oracle Enterprise Linux 5 64-bit guest, add the following to the virt-install command-line:

          # virt-install hvm ... os-type=linux --os-variant=el5_64 ...
          For best results, additional parameters may be needed in the boot loader (grub.conf) configuration file for certain operating system variants after the guest is installed. Specifically, for optimal clock accuracy, Linux guest boot parameters should be specified to ensure that the pit clock source is utilized. Adding clock=pit nohpet nopmtimer for most guests will result in the selection of pit as the clock source for the guest. Published templates for Oracle VM will include these additional parameters.

          Proper maintenance of virtual time can be tricky. The various parameters provide tuning for virtual time management and supplement, but do not replace, the need for an ntp time service running within guest. Ensure that the ntpd service is running and that the /etc/ntpd.conf configuration file is pointing to valid time servers.

          Kirill Babeyev
          • 2. Re: OEL6 System Always Boots w/ Bad Time
            Rob
            Krill,

            Does that apply to PVM systems, as well? I don't have any HVM systems in my OVM environment.

            Rob
            • 3. Re: OEL6 System Always Boots w/ Bad Time
              929328
              Hi Rob,

              This is quite another situation. By default PVM Xen guests try and synchronise their clock with that of their hypervisor so one thing to check is what the "local" time is. The dance for configuring the OVM manager as the NTP server is rather error prone.

              We get much better results by disabling this synchronisation in RedHat 5 guests via /etc/sysctl.conf

              xen.independent_wallclock = 1

              This means that the guests will then rely on their own ntp daemon and the kernel tick from the hypervisor to keep the correct time.

              Ask if you have questions.
              Kirill Babeyev
              • 4. Re: OEL6 System Always Boots w/ Bad Time
                Rob
                Krill,

                Still no luck. I set the option, rebooted, used ntpdate to set the time, rebooted, and it came back up 7 hours slow again (we're at UTC-7 here).

                I do see this clock entry in the logs:

                Aug 1 03:07:49 ovm-cc kernel: Switching to clocksource xen

                Is that accurate?

                Rob
                • 5. Re: OEL6 System Always Boots w/ Bad Time
                  929328
                  Rob,
                  Try this code:
                  [root@OracleVM ~]# hwclock
                  Cannot access the Hardware Clock via any known method.
                  Use the --debug option to see the details of our search for an access method.
                  [root@OracleVM ~]# ls -al /dev/rt*
                  crw------- 1 root root 252, 0 Jan 25 10:25 /dev/rtc0
                  [root@OracleVM ~]# ln -s /dev/rtc0 /dev/rtc
                  [root@OracleVM ~]# ls -al /dev/rt*
                  lrwxrwxrwx 1 root root 9 Jan 25 18:18 /dev/rtc -> /dev/rtc0
                  crw------- 1 root root 252, 0 Jan 25 10:25 /dev/rtc0
                  [root@OracleVM ~]# hwclock
                  Wed 25 Jan 2012 11:19:00 PM EST -0.369001 seconds
                  [root@OracleVM ~]#


                  (That's all you should get in the end)
                  Ask if you have questions.
                  Kirill Babeyev
                  • 6. Re: OEL6 System Always Boots w/ Bad Time
                    Rob
                    Krill,

                    [root@ovm-utility-db ~]# ls -al /dev/rt*
                    ls: cannot access /dev/rt*: No such file or directory

                    No dice. :-(

                    As this is now impacting more machines, I submitted an SR to Oracle. They're telling me to use NTP to compensate... hardly seems worth the price of admission.

                    Rob
                    • 7. Re: OEL6 System Always Boots w/ Bad Time
                      Terry Phelps
                      I haven't heard anyone ask whether the time zone in the machine is set properly. Is it?

                      Is the time almost exactly 7 hours off? If so, that's not "clock skew", due to virtualization effects. If the time is very nearly 7 hours off, then I gotta think it's keeping good time, in the sense that it's tracking UTC well, but it's simply displaying UTC time instead of local time.

                      When you use the "date" command in Unix, it displays what it think the user's local time zone is. What does you /etc/sysconfig/clock say?

                      Mine says:
                      ZONE="America/Kentucky/Louisville"
                      UTC=true
                      ARC=false

                      You should have ZONE set to whatever the mnemonic for your time zone is.
                      • 8. Re: OEL6 System Always Boots w/ Bad Time
                        Rob
                        Terry,

                        The time zone is set properly (to Los Angeles), and the clock is almost exactly 7 hours off... and we're at UTC-7 currently.

                        The hardware clock on the virtual servers are set to local time (not UTC), but it seems to be carrying over to some of the VMs as being set to UTC. The time zone is set properly on the virtual servers, as well...

                        Rob
                        • 9. Re: OEL6 System Always Boots w/ Bad Time
                          Terry Phelps
                          You said Oracle's response to your SR was "just use NTP to fix it". You're seeing what I see whenever I open an SR against OVM -- they're utterly useless.

                          The fact that ntpdate fixes the problem, and then the problem stays fixed after that (right?), tells me that the machine keeps good time. It's just that it's confused about what the time IS after a reboot. If you have nothing better to try, I suggest the following:

                          Reboot and get the bad time.

                          (Humor me and) make real, real sure the time zone is right. Do this in a shell:

                          date
                          # export TZ=PST8PDT
                          date

                          If date shows the same time (almost) in both cases, I'll be convinced.

                          I'm no expert on hwclock and such, but surely that must be involved.
                          Search google for "linux clock always wrong after reboot", without the quotes, and look over the top few hits
                          Read over the hwclock manpage.
                          Look at your /etc/adjtime file, and figure out what it all means.
                          What happens if you rename the adjtime file to something else, and reboot?

                          I'll be interested to here the solution when you find it.
                          • 10. Re: OEL6 System Always Boots w/ Bad Time
                            Alan3
                            I had intermittent issues with VM clocks for quite some time...
                            When I've ran into this, I just add a cron task to keep the time in sync with a local ntp server...

                            * * * * * /usr/sbin/ntpdate [your ntp server address here] > /dev/null 2>&1

                            Your mileage may vary - I've never had any problems doing this...
                            • 11. Re: OEL6 System Always Boots w/ Bad Time
                              J Peters-Oracle
                              Defining "UTC" in "/etc/sysconfig/clock" is deprecated in OL6.

                              Excerpt "/usr/share/doc/initscripts-9.03.23/sysconfig.txt",

                              /etc/sysconfig/clock:

                              Current releases use the third parameter in the /etc/adjtime
                              file (either 'UTC' or 'LOCAL') to determine whether the hwclock
                              is in UTC or localtime.

                              ZONE="filename" indicates the zonefile under /usr/share/zoneinfo
                              that /etc/localtime is a copy of, for example:
                              ZONE="US/Eastern"

                              deprecated values from earlier releases:

                              CLOCKMODE=GMT indicates that the clock is set to UTC
                              CLOCKMODE=ARC on alpha only indicates the ARC console's
                              42-year time offset is in effect
                              UTC=true,yes
                              Indicates that the hardware clock is set to UTC.
                              UTC=no,false
                              Indicates that the hardware clock is set to Local Time.
                              ARC=true on alpha only indicates the ARC console's
                              42-year time offset is in effect; otherwise the normal
                              Unix epoch is assumed.

                              SRM=true on alpha only indicates the SRM 1900 epoch is in
                              effect; otherwise the normal Unix epoch is assumed.
                              • 12. Re: OEL6 System Always Boots w/ Bad Time
                                J Peters-Oracle
                                FYI you do not have "hwclock" if the OL6 VM is using PV.

                                OL6u1 HVM,

                                [root@OL6u1-hvm ~]# cat /etc/issue
                                Oracle Linux Server release 6.1
                                Kernel \r on an \m

                                [root@OL6u1-hvm ~]# hwclock
                                Wed 17 Oct 2012 03:44:45 PM MDT -0.147479 seconds

                                OL6u1 PV,

                                [root@OL6u1-pv ~]# cat /etc/issue
                                Oracle Linux Server release 6.1
                                Kernel \r on an \m

                                [root@OL6u1-pv ~]# hwclock
                                Cannot access the Hardware Clock via any known method.
                                Use the --debug option to see the details of our search for an access method.