12 Replies Latest reply on Oct 1, 2007 7:30 PM by user4994457

    defunct process

    807559
      Hi all

      I found a lot of defunct process in one of our solaris server. It is occupying around 95% of the memory.I killed all those defunct processes, but memory is not yet released.Anybody have an idea about how to release that memory without rebooting the system


      Regards
      RPS
        • 1. Re: defunct process
          abrante
          If the processes are still there you could try preap, but the memory might still not be freed up if its somehow allocated by the defunct's process parent. Still worth a try.

          See:
          man preap

          7/M.
          • 2. Re: defunct process
            user4994457
            Hi all

            I found a lot of defunct process in one of our
            solaris server. It is occupying around 95% of the
            memory.
            What is 'it'? Defunct processes are dead and don't occupy any memory.

            What leads you to believe that the defunct processes are responsible?
            I killed all those defunct processes, but
            memory is not yet released.
            You can't kill defunct processes, they're already dead. All you can do is convince their parent to reap their exit status, or kill the parent so that 'init' does the job.
            Anybody have an idea about
            how to release that memory without rebooting the
            system
            What method are you using to deermine memory in use? Is this a Solaris 9 machine? Are you using 'ps' and looking at the virtual memory size?

            --
            Darren
            • 3. Re: defunct process
              807559
              I am able to kill those defunct process

              ps -ef |grep defunct|awk '{print $2}'|xargs kill -9


              and i am able to see the memory using prstat -a -s size
              • 4. Re: defunct process
                user4994457
                I am able to kill those defunct process
                You have a very different Solaris system than I do then. Are you sure they're not just getting reaped between when you see them and when you look after the kill? Can you show a 'ps' output for them?

                # ps -ef | grep defu
                root 3199 3198 0 0:00 <defunct>
                # kill -9 3199
                # ps -ef | grep defu
                root 3199 3198 0 0:00 <defunct>
                and i am able to see the memory using prstat -a -s
                size
                We can look at it a litte more directly with ps. Can you try this on one of your defunct processes and show the output?

                # ps -p 3199 -o pid,s,vsz,rss,args
                PID S VSZ RSS COMMAND
                3199 Z 0 0

                Notice that on this machine a defunct process occupies no virtual memory and also no RAM pages.

                --
                Darren
                • 5. Re: defunct process
                  807559
                  Thanks, you are exactly right.

                  you mean to say dufunct process is not responsible for the performance degade of the system?
                  • 6. Re: defunct process
                    807559
                    Not really no. The only thing a zombie/defunct occupies is a process ID (PID). So you -could- run out of those. But things like memory and such should've been relinquished to the system already.
                    • 7. Re: defunct process
                      807559
                      I am not part of this 'fight'/. If you look at the man page of preap, you come to know that zombies consume a bit of system memory. (second para, last line)

                      If you have thosands such processes, why system performance can't be hindered??
                      • 8. Re: defunct process
                        user4994457
                        Yup. I believe the "small amount of system memory" that it mentions is in fact the allocation for the exit status.

                        You will almost certainly run out of PIDs before you notice the memory consumed by the zombies.

                        Heck, I've got a box here I can abuse, let's try it. It's even painfully low on memory, being an Ultra-10 with 192MB and Solaris 10 and I'm logged in with a JDS desktop. "max_nprocs" on this machine is 2922.

                        I ran this command:
                        # perl -e 'while (1) { $a = fork(); if (!defined $a) {print "Unable to fork.  $! Sleeping.\n" ; sleep 30; exit; } elsif ($a == 0) { $b = 7 ; $b++; exit; } }'

                        (The math with $b is just there so that the child does something besides exit, hopefully forcing at least one page allocation that should be returned).

                        When it finished, 'prstat' went from 75 processes to 2919 (pretty close to 2922).

                        vmstat went crazy during the run (and load was over 50 temporarily), but it settled down.

                        0 0 0 413928 34360 0 52 0 0 0 0 0 0 0 0 0 406 979 402 6 4 90
                        [job started here...]
                        42 0 0 397208 29632 155 3871 117 0 0 0 0 15 0 0 0 440 1058 363 13 46 41
                        134 0 0 368048 30256 253 6847 37 335 447 0 237 7 0 0 0 415 1540 324 18 82 0
                        [ran a while... and settled down during the 30 second sleep]
                        0 0 0 413472 35760 0 52 0 0 0 0 0 0 0 0 0 401 964 249 5 3 92
                        0 0 0 413472 35760 0 52 0 0 0 0 0 0 0 0 0 402 984 266 4 3 92
                        [parent exits and all the zombies are reaped]
                        0 0 0 413800 36736 0 53 0 0 0 0 0 0 0 0 0 402 3267 250 6 16 78
                        0 0 0 413888 36984 0 52 0 0 0 0 0 1 0 0 0 406 901 258 4 3 93

                        So, yes. A bit of memory was returned. For over 2800 zombies, something on the order of 400 kb was returned on exit. That's about 140 bytes per process.

                        'vmstat' isn't necessarily completely accurate and something else on the machine may have affected the output. Assume that the above could be off by 100% or more and it's still not a signifcant amount..

                        --
                        Darren
                        • 9. Re: defunct process
                          807559
                          Thanks for all your response

                          Any way i am able to kill 99% of defunct processes in one of my servers
                          using the following command

                          ps -ef |grep defunct|awk '{print $2}'|xargs preap -F
                          preap: cannot examine 14145: no such process
                          preap: waitid() in process 15897 failed: Operation not applicable3821: exited with status 0
                          10335: exited with status 0
                          4183: exited with status 0
                          9075: exited with status 0
                          12615: exited with status 0
                          6661: exited with status 0
                          21295: exited with status 0
                          21861: exited with status 0
                          22031: exited with status 0
                          13050: exited with status 0
                          2870: exited with status 0
                          21949: exited with status 0
                          17599: exited with status 0
                          29764: exited with status 0
                          15154: exited with status 0
                          28853: exited with status 0
                          9639: exited with status 0
                          21735: exited with status 0
                          24742: exited with status 0
                          9458: exited with status 0
                          29567: exited with status 0
                          4779: exited with status 0
                          14898: exited with status 0
                          12881: exited with status 0
                          6946: exited with status 0
                          12905: exited with status 0
                          15795: exited with status 0
                          • 10. Re: defunct process
                            abrante
                            Well, zombies are afterall in the proc tables etc, i guess that will occupy a (very) small piece of the memory ;)

                            Btw, when using "ps" and piping the results to "grep", its good to add "grep -v grep".. :-)



                            7/M.
                            • 11. Re: defunct process
                              807559
                              Hi
                              I am working on solaris 10 and i am facing problem of *<defunct>* process.i try to kill for this command ,*# kill -9 PID* but this command not able kill and also not release memory.

                              What can i do for this *<defunct>* process.

                              My mail Id is alok7.k@tcs.com

                              Edited by: alok7.k on Sep 29, 2007 7:51 AM
                              • 12. Re: defunct process
                                user4994457
                                Defunct processes are already dead. You cannot kill them.

                                The parent that spawned the process needs to wait() on it to gather the exit status and allow the process to disappear, or the parent has to exit.

                                If you do 'ps -f -p <PID>', you'll see the parent's PID under 'PPID'. That process is responsible for the zombie.

                                If that process is '1', something is very wrong.

                                --
                                Darren