3 Replies Latest reply: Nov 12, 2011 9:14 PM by Mgerdts-Oracle RSS

    Solaris 10 Zones (global/local) /proc fs is 97% full

    ostendali
      Hi Folks,
      sorry in case of this thread has been discussed already but I couldn't find any info regardless.

      I've a solaris 10 server on which have installed 3 zones and 2 of these zone giving me continuously alert regarding disk space.

      I know when /proc fs is full is normal but in my case when I am looking into disk space of solaris zone:

      -bash-3.00# df -ak
      Filesystem 1024-blocks Used Available Capacity Mounted on
      / 70220321 67141358 2376760 97% /
      /dev 70220321 67141358 2376760 97% /dev
      proc 0 0 0 0% /proc
      ctfs 0 0 0 0% /system/contract
      mnttab 0 0 0 0% /etc/mnttab
      objfs 0 0 0 0% /system/object
      swap 18973688 224 18973464 1% /etc/svc/volatile
      /platform/SUNW,Sun-Fire-T200/lib/libc_psr/libc_psr_hwcap1.so.1
      70220321 67141358 2376760 97% /platform/sun4v/lib/libc_psr.so.1
      /platform/SUNW,Sun-Fire-T200/lib/sparcv9/libc_psr/libc_psr_hwcap1.so.1
      70220321 67141358 2376760 97% /platform/sun4v/lib/sparcv9/libc_psr.so.1
      fd 0 0 0 0% /dev/fd
      swap 19074320 100856 18973464 1% /tmp
      swap 18973488 24 18973464 1% /var/run
      -hosts 0 0 0 0% /net


      As my understanding this issue filling up / fs as well, which totally strange to me.

      Could someone advise pls why this mess?

      Thanks
        • 1. Re: Solaris 10 Zones (global/local) /proc fs is 97% full
          abrante
          Well, as mentioned something is filling up your root (/) filesystem, you will have to find the cause of this issue by running du.

          Most commonly the thing which is filling up is under /var, but you can find it out with du.

          I.e:
          cd /
          du -hs *

          .7/M.
          • 2. Re: Solaris 10 Zones (global/local) /proc fs is 97% full
            ostendali
            Thanks for reply.

            I've already ran du command, sorry for haven't posted here:


            -bash-3.00# du -ash /* |sort -nr |grep G
            870G /proc
            51G /vhosts
            -bash-3.00# ls /proc/
            1118 1214 1263 1447 6701 752 7826 7908 7913 7948 8044
            1140 1229 1266 1448 713 7614 7843 7910 7914 7951 945
            1201 1234 1269 1561 735 7635 7902 7911 7915 7952 947
            1213 12375 1343 3290 749 7636 7903 7912 7916 8029
            -bash-3.00# du -ash /proc/* |sort -nr |grep G
            du: /proc/7636/path/145: No such file or directory
            du: /proc/7843: No such file or directory
            du: /proc/7903: No such file or directory
            46G /proc/8056
            46G /proc/8055
            46G /proc/8054
            46G /proc/8029
            46G /proc/7952
            46G /proc/7951
            46G /proc/7948
            46G /proc/7902
            46G /proc/7826
            46G /proc/7636
            46G /proc/7635
            46G /proc/6701
            46G /proc/3290
            -bash-3.00# ps -ef |grep 8056
            root 8288 7614 0 06:04:52 pts/7 0:00 grep 8056
            -bash-3.00# cat /proc/805
            8054/ 8055/
            -bash-3.00# ps -ef |grep 8055
            root 8467 7614 0 06:05:33 pts/7 0:00 grep 8055
            daemon 8055 3290 0 06:04:09 ? 0:02 /usr/local/httpd-php-2.2.11/bin/httpd -k start
            -bash-3.00# ps -ef |grep 8054
            root 8472 7614 0 06:05:42 pts/7 0:00 grep 8054
            daemon 8054 3290 0 06:04:08 ? 0:01 /usr/local/httpd-php-2.2.11/bin/httpd -k start
            -bash-3.00# ps -ef |grep 8029
            daemon 8029 3290 0 06:03:20 ? 0:01 /usr/local/httpd-php-2.2.11/bin/httpd -k start
            root 8478 7614 0 06:05:57 pts/7 0:00 grep 8029
            -bash-3.00# ps -ef |grep 7952
            daemon 7952 3290 0 06:02:41 ? 0:01 /usr/local/httpd-php-2.2.11/bin/httpd -k start
            root 8485 7614 0 06:06:13 pts/7 0:00 grep 7952


            As you can see the fs is filled up from processes /proc and vhost

            I've investigated and noticed that related to apache instances....

            I could't figure out why.......

            here the output by checking inode usage in fs:

            -bash-3.00# df -F ufs -o i
            Filesystem iused ifree %iused Mounted on
            / 799896 7650728 9% /

            No clue what is going on....

            Thanks in advance for any reply....
            • 3. Re: Solaris 10 Zones (global/local) /proc fs is 97% full
              Mgerdts-Oracle
              ostendali wrote:
              Thanks for reply.

              I've already ran du command, sorry for haven't posted here:


              -bash-3.00# du -ash /* |sort -nr |grep G
              A more useful command is:

              du -kd / | sort -n | tail -30

              Why is it better?

              - The "/" argument tells it to look at /. Your command says "look at all the stuff that is in /*". Note that /* does not include files and/or directories that start with ".". It does include file systems that are mounted on subdirectories of /.
              - The -k option makes it so all of the units are the same. Thus, sort works better and you have an easy visual of the difference in space by the number of digits in the size.
              - The -d option makes it so that it doesn't look in other file systems. You are trying to figure out why / is full. /proc is a different file system that is a representation of a bunch of kernel data structures. /proc takes no space on disk.
              As you can see the fs is filled up from processes /proc and vhost
              /proc is not filling up any disk-based file system. When you read a file that is in /proc, you are either reading a representation of kernel data structures that are stored in memory or you are being referred off to a file in a different file system.

              Notice that between your posts, you show that the root file system is about 70 GB in total size and /vhosts seems to be using 51 GB. That means that you are looking for less than 20 GB. If the du command I give above shows that / is using a lot less than df says it is using, try this command:

              # find /proc/*/fd -type f -links 0 | xargs du -k | sort -n
              ...
              258     /proc/12328/fd/24

              Each line represents a file that has been removed but something still has it open. Because it is removed (no longer "linked" into a directory) it is invisible to du when traversing the normal file system. /proc provides a special view into it. In my example pid 12328 has file descriptor 24 open to a removed file. It is 258 KB in size. When pid 12328 dies, the space used by that file will be freed.

              You can typically see what is in such a file using a command like

              # less /proc/12328/fd/24

              Many times you will see that it is a log file that is being written to and will never be read (because it is removed, after all). In such cases, you can free the space immediately with:

              # cp /dev/null /proc/12328/fd/24