6 Replies Latest reply: Dec 29, 2013 7:29 AM by user12164083 RSS

    / (root) partition %100 full

    user12164083

      Hi,

      on oracle Linux 5.8 64 bit,

       

      strangely my root (/) partition is %100 full. when i check folders under / there is no big size folder.

       

       

       

      Filesystem            Size  Used Avail Use% Mounted on

      /dev/sda3              33G   33G     0 100% /

      /dev/sda1             1.9G   52M  1.8G   3% /boot

      tmpfs                  30G     0   30G   0% /dev/shm

      /dev/sdb1             788G   91G  658G  13% /data1

      /dev/sdc1             788G   87G  661G  12% /data2

      /dev/sdd1             788G   86G  663G  12% /data3

      /dev/sde1             640G   82G  527G  14% /data4

      /dev/sdf1             296G   19G  263G   7% /data5

      /dev/sdg1             247G   87G  148G  37% /archive

      172.16.16.22:/data/col1/tetasdb1

                            7.7T  3.8T  3.9T  50% /datadomain

      172.16.16.98:/VOLUME1/BANKA

                            1.7T  803G  873G  48% /BANKALAR

       

      as you can see in du -ksh * command output (bolded partition mount points) there is no big size folder under /

       

      [root@xxx /]# du -ksh *

      86G     archive

      5.2G    BANKALAR

      8.4M    bin

      17M     boot

      91G     data1

      87G     data2

      86G     data3

      81G     data4

      18G     data5

      559G    datadomain

      260K    dev

      115M    etc

      29M     home

      4.0K    kenan.txt

      246M    lib

      30M     lib64

      16K     lost+found

      8.0K    media

      0       misc

      8.0K    mnt

      0       net

      546M    opt

      0       orahome

      0       proc

      1.5M    root

      38M     sbin

      8.0K    selinux

      8.0K    srv

      0       sys

      100K    tftpboot

      56K     tmp

      4.1G    usr

      133M    var

      4.0K    vdp

       

       

      I need your help urgently.

        • 1. Re: / (root) partition %100 full
          Catch~22

          Just and idea, but Unix or Linux does not prevent you from overlaying a non empty directory with another file system. Perhaps your mount points were not empty directories. Under such circumstances, the du command will not show the data.

           

          For example:

           

          Terminal

          [root@vm210 /]# df -Ph /

          Filesystem            Size  Used Avail Use% Mounted on

          /dev/mapper/vg_vm210-lv_root   16G  1.9G   13G  13% /


          [root@vm210 /]# du -ksh /

          1.8G    /

           

          Create a 1 GB file in /test:

           

          Terminal

          [root@vm210 /]# mkdir /test

          [root@vm210 /]# dd if=/dev/zero of=/test/testfile bs=1G count=1

          1+0 records in

          1+0 records out

          1073741824 bytes (1.1 GB) copied, 2.16613 s, 496 MB/s


          [root@vm210 /]# df -Ph /

          Filesystem            Size  Used Avail Use% Mounted on

          /dev/mapper/vg_vm210-lv_root   16G  2.9G   12G  20% /


          [root@vm210 /]# du -ksh /

          2.8G    /

           

          Mount another file system under /test:

           

          Terminal

          [root@vm210 /]# parted -s /dev/sdb mklabel msdos mkpart primary 1 -- -1

          [root@vm210 /]# mkfs.ext4 -L testdrive /dev/sdb1

          [root@vm210 /]# mount /dev/sdb1 /test


          [root@vm210 /]# df -Ph /

          Filesystem            Size  Used Avail Use% Mounted on

          /dev/mapper/vg_vm210-lv_root   16G  2.9G   12G  20% /


          [root@vm210 /]# du -ksh /

          1.8G    /

          • 2. Re: / (root) partition %100 full
            user12164083

            Hi Dude,

             

            yes you are right but i do not think this is the case because we have this problem for a week.

             

            this server is up for 6 months.

             

            this problem makes me crazy because somehow we must be able to find out what is causing this problem and what is fulling disk.

            • 3. Re: / (root) partition %100 full
              Catch~22

              You may have a hidden file or directory, starting with a . (dot).

               

              Check the following:

               

               

              Terminal

              [root@vm210 ~]# ll .testfile

              -rw-r--r--. 1 root root 1073741824 Mar 15 12:55 .testfile

               

              [root@vm210 ~]# du -ksh

              1.1G    .


              [root@vm210 ~]# du -ksh .*

              1.1G    .

               

               

               

              [root@vm210 ~]# du -ksh *

              4.0K    anaconda-ks.cfg

              32K     install.log

              8.0K    install.log.syslog

              • 4. Re: / (root) partition %100 full
                user12164083

                Hi Dude;

                i moved ibm tivoli folder that is under /opt to another partition and got some free space.

                 

                but yesterday root(/) partition became full again.

                 

                what the problem we had yesterday was:

                there is a NFS mount point on host and because NFS host was unreachable, we could not even run df -h command.

                 

                after solving problem on NFS host, mounted it successfully but ther root ( / ) is still full.

                 

                do you think NFS mount problem caused this issue?

                 

                if so what is fulling it?

                • 5. Re: / (root) partition %100 full
                  beb7a7cd-ba2e-48b6-a17e-336098294e1d

                  Did you check the log files in the server ?

                  • 6. Re: / (root) partition %100 full
                    user12164083

                    hi, in fact dude's answer is correct.

                     

                    there is a local directory /datadomain/Archive.

                     

                    also there is a nfs mount point /datadomain and under nfs there is Archive directory.

                     

                    a scheduled script copies archive logs to this nfs mount directory /datadomain/Archive hourly.

                     

                    bu if this nfs mount point is not mounted, script continues to copy archive logs to  /datadomain/Archive directory which exists on / (root partition).

                     

                    unmounted /datadomain (nfs)

                     

                    and removed directory Archive under /datadomain

                     

                    and then mount /datadomain (NFS) again.

                     

                     

                    problem solved.