6 Replies Latest reply: Dec 29, 2013 5:29 AM by user12164083 RSS

/ (root) partition %100 full

user12164083 Newbie
Currently Being Moderated

Hi,

on oracle Linux 5.8 64 bit,

 

strangely my root (/) partition is %100 full. when i check folders under / there is no big size folder.

 

 

 

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda3              33G   33G     0 100% /

/dev/sda1             1.9G   52M  1.8G   3% /boot

tmpfs                  30G     0   30G   0% /dev/shm

/dev/sdb1             788G   91G  658G  13% /data1

/dev/sdc1             788G   87G  661G  12% /data2

/dev/sdd1             788G   86G  663G  12% /data3

/dev/sde1             640G   82G  527G  14% /data4

/dev/sdf1             296G   19G  263G   7% /data5

/dev/sdg1             247G   87G  148G  37% /archive

172.16.16.22:/data/col1/tetasdb1

                      7.7T  3.8T  3.9T  50% /datadomain

172.16.16.98:/VOLUME1/BANKA

                      1.7T  803G  873G  48% /BANKALAR

 

as you can see in du -ksh * command output (bolded partition mount points) there is no big size folder under /

 

[root@xxx /]# du -ksh *

86G     archive

5.2G    BANKALAR

8.4M    bin

17M     boot

91G     data1

87G     data2

86G     data3

81G     data4

18G     data5

559G    datadomain

260K    dev

115M    etc

29M     home

4.0K    kenan.txt

246M    lib

30M     lib64

16K     lost+found

8.0K    media

0       misc

8.0K    mnt

0       net

546M    opt

0       orahome

0       proc

1.5M    root

38M     sbin

8.0K    selinux

8.0K    srv

0       sys

100K    tftpboot

56K     tmp

4.1G    usr

133M    var

4.0K    vdp

 

 

I need your help urgently.

  • 1. Re: / (root) partition %100 full
    Dude! Guru
    Currently Being Moderated

    Just and idea, but Unix or Linux does not prevent you from overlaying a non empty directory with another file system. Perhaps your mount points were not empty directories. Under such circumstances, the du command will not show the data.

     

    For example:

     

    Terminal

    [root@vm210 /]# df -Ph /

    Filesystem            Size  Used Avail Use% Mounted on

    /dev/mapper/vg_vm210-lv_root   16G  1.9G   13G  13% /


    [root@vm210 /]# du -ksh /

    1.8G    /

     

    Create a 1 GB file in /test:

     

    Terminal

    [root@vm210 /]# mkdir /test

    [root@vm210 /]# dd if=/dev/zero of=/test/testfile bs=1G count=1

    1+0 records in

    1+0 records out

    1073741824 bytes (1.1 GB) copied, 2.16613 s, 496 MB/s


    [root@vm210 /]# df -Ph /

    Filesystem            Size  Used Avail Use% Mounted on

    /dev/mapper/vg_vm210-lv_root   16G  2.9G   12G  20% /


    [root@vm210 /]# du -ksh /

    2.8G    /

     

    Mount another file system under /test:

     

    Terminal

    [root@vm210 /]# parted -s /dev/sdb mklabel msdos mkpart primary 1 -- -1

    [root@vm210 /]# mkfs.ext4 -L testdrive /dev/sdb1

    [root@vm210 /]# mount /dev/sdb1 /test


    [root@vm210 /]# df -Ph /

    Filesystem            Size  Used Avail Use% Mounted on

    /dev/mapper/vg_vm210-lv_root   16G  2.9G   12G  20% /


    [root@vm210 /]# du -ksh /

    1.8G    /

  • 2. Re: / (root) partition %100 full
    user12164083 Newbie
    Currently Being Moderated

    Hi Dude,

     

    yes you are right but i do not think this is the case because we have this problem for a week.

     

    this server is up for 6 months.

     

    this problem makes me crazy because somehow we must be able to find out what is causing this problem and what is fulling disk.

  • 3. Re: / (root) partition %100 full
    Dude! Guru
    Currently Being Moderated

    You may have a hidden file or directory, starting with a . (dot).

     

    Check the following:

     

     

    Terminal

    [root@vm210 ~]# ll .testfile

    -rw-r--r--. 1 root root 1073741824 Mar 15 12:55 .testfile

     

    [root@vm210 ~]# du -ksh

    1.1G    .


    [root@vm210 ~]# du -ksh .*

    1.1G    .

     

     

     

    [root@vm210 ~]# du -ksh *

    4.0K    anaconda-ks.cfg

    32K     install.log

    8.0K    install.log.syslog

  • 4. Re: / (root) partition %100 full
    user12164083 Newbie
    Currently Being Moderated

    Hi Dude;

    i moved ibm tivoli folder that is under /opt to another partition and got some free space.

     

    but yesterday root(/) partition became full again.

     

    what the problem we had yesterday was:

    there is a NFS mount point on host and because NFS host was unreachable, we could not even run df -h command.

     

    after solving problem on NFS host, mounted it successfully but ther root ( / ) is still full.

     

    do you think NFS mount problem caused this issue?

     

    if so what is fulling it?

  • 5. Re: / (root) partition %100 full
    beb7a7cd-ba2e-48b6-a17e-336098294e1d Newbie
    Currently Being Moderated

    Did you check the log files in the server ?

  • 6. Re: / (root) partition %100 full
    user12164083 Newbie
    Currently Being Moderated

    hi, in fact dude's answer is correct.

     

    there is a local directory /datadomain/Archive.

     

    also there is a nfs mount point /datadomain and under nfs there is Archive directory.

     

    a scheduled script copies archive logs to this nfs mount directory /datadomain/Archive hourly.

     

    bu if this nfs mount point is not mounted, script continues to copy archive logs to  /datadomain/Archive directory which exists on / (root partition).

     

    unmounted /datadomain (nfs)

     

    and removed directory Archive under /datadomain

     

    and then mount /datadomain (NFS) again.

     

     

    problem solved.

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points