7 Replies Latest reply: Feb 27, 2014 10:16 AM by Catch-22 RSS

    Processes showing huge stime(s) even after reboot

    user1024842

      [root@oeldb9 ~]#uname -a
      Linux oeldb9 2.6.39-200.24.1.el6uek.x86_64 #1 SMP Sat Jun 23 02:39:07 EDT 2012 x86_64 x86_64 x86_64 GNU/Linux

       

       

      [root@oeldb9 ~]#cat /etc/redhat-release
      Red Hat Enterprise Linux Server release 6.3 (Santiago)

       

       

      [root@oeldb9 ~]#uptime
      09:01:43 up 2 days, 14:49, 11 users,  load average: 16.39, 11.62, 10.22

       

      Boot time is correct

      [root@oeldb9 ~]#cat /proc/stat |grep btime
      btime 1393204316

       

      Init shows running 625+ days though it was rebooted on Monday

      [root@oeldb9 ~]#cat /proc/1/stat | awk '{print $15}'
      5121045649

      [root@oeldb9 ~]#ps -ef|grep init

      root         1     0 99 Feb23 ?        625-13:23:31 /sbin/init

       

      On a similar host

      [root@oeldb11 ~]#cat /proc/1/stat | awk '{print $15}'
      682

       

      Some processes show 100s of thousands of days ( though just started this morning.

      oracle    1906     1 99 07:18 ?        213503-23:35:35 <name omitted> (LOCAL=NO)

      [root@oeldb9 ~]#cat /proc/1906/stat | awk '{print $15}'
      1842911911730

       

       

      Has anyone else seen this?

        • 1. Re: Processes showing huge stime(s) even after reboot
          Paul In 'T Hout-Oracle

          I have not seen this before, very strange.

          You have probably verified the following already, but just to make sure. Did you completely powerdown the machine for the reboot, or was it suspended / hibernated and then returned to live?

          It's my understanding that /proc should be completely re-established at reboot time.

          • 2. Re: Processes showing huge stime(s) even after reboot
            Catch-22

            Just a thought.... 213503 seconds are 2.47 days. Can you see if that number increases by seconds?

             

            Btw, how many CPU's are in the machine?

            • 3. Re: Processes showing huge stime(s) even after reboot
              Paul In 'T Hout-Oracle

              does init show high up there in 'top'? perhaps it's an idea to do an strace on it to see what it is doing exactly

              • 4. Re: Processes showing huge stime(s) even after reboot
                user1024842

                Complete reboot, I watched it from the console.   The timestamps themselves on the /proc entries are just fine.  It is the stime for stat that are off the charts.

                • 5. Re: Processes showing huge stime(s) even after reboot
                  user1024842

                  32 Physical cores, HT turned on, so /proc/cpuinfo has 0-63 ( 64 ).

                  • 6. Re: Processes showing huge stime(s) even after reboot
                    user1024842

                    init is NOT near the top of "top/htop"

                     

                    Here is a quick strace -p1 output.

                     

                    write(4, "\0", 1)                       = 1
                    rt_sigreturn(0x4)                       = -1 EINTR (Interrupted system call)
                    read(3, "\0", 1)                        = 1
                    read(3, 0x7fff8670743f, 1)              = -1 EAGAIN (Resource temporarily unavailable)
                    waitid(P_ALL, 0, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19436, si_status=0, si_utime=0, si_stime=0}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
                    waitid(P_ALL, 0, {}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
                    select(10, [3 5 6 7 9], [], [7 9], NULL) = ? ERESTARTNOHAND (To be restarted)
                    --- SIGCHLD (Child exited) @ 0 (0) ---
                    write(4, "\0", 1)                       = 1
                    rt_sigreturn(0x4)                       = -1 EINTR (Interrupted system call)
                    read(3, "\0", 1)                        = 1
                    read(3, 0x7fff8670743f, 1)              = -1 EAGAIN (Resource temporarily unavailable)
                    waitid(P_ALL, 0, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19432, si_status=0, si_utime=0, si_stime=0}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
                    waitid(P_ALL, 0, {}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
                    select(10, [3 5 6 7 9], [], [7 9], NULL) = ? ERESTARTNOHAND (To be restarted)
                    --- SIGCHLD (Child exited) @ 0 (0) ---
                    write(4, "\0", 1)                       = 1
                    rt_sigreturn(0x4)                       = -1 EINTR (Interrupted system call)
                    read(3, "\0", 1)                        = 1
                    read(3, 0x7fff8670743f, 1)              = -1 EAGAIN (Resource temporarily unavailable)
                    waitid(P_ALL, 0, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19503, si_status=0, si_utime=0, si_stime=0}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
                    waitid(P_ALL, 0, {}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
                    select(10, [3 5 6 7 9], [], [7 9], NULL) = ? ERESTARTNOHAND (To be restarted)
                    --- SIGCHLD (Child exited) @ 0 (0) ---
                    write(4, "\0", 1)                       = 1
                    rt_sigreturn(0x4)                       = -1 EINTR (Interrupted system call)
                    read(3, "\0", 1)                        = 1
                    read(3, 0x7fff8670743f, 1)              = -1 EAGAIN (Resource temporarily unavailable)
                    waitid(P_ALL, 0, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19511, si_status=0, si_utime=0, si_stime=0}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
                    waitid(P_ALL, 0, {}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
                    select(10, [3 5 6 7 9], [], [7 9], NULL) = ? ERESTARTNOHAND (To be restarted)
                    --- SIGCHLD (Child exited) @ 0 (0) ---
                    write(4, "\0", 1)                       = 1
                    rt_sigreturn(0x4)                       = -1 EINTR (Interrupted system call)
                    read(3, "\0", 1)                        = 1
                    read(3, 0x7fff8670743f, 1)              = -1 EAGAIN (Resource temporarily unavailable)
                    waitid(P_ALL, 0, {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=19520, si_status=0, si_utime=0, si_stime=0}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
                    waitid(P_ALL, 0, {}, WNOHANG|WEXITED|WSTOPPED|WCONTINUED, NULL) = 0
                    select(10, [3 5 6 7 9], [], [7 9], NULL^C <unfinished ...>

                    • 7. Re: Processes showing huge stime(s) even after reboot
                      Catch-22

                      Is this a virtual machine? If yes, the problem might be there. I also suggest to verify you have installed the latest BIOS for your machine. Perhaps you can try the UEK3 kernel to see if the problem persists.