Root Filesystem Suddenly Full

[damo12]

Hi,

I support a company running a small Ubuntu server. It is running Ubuntu 10.0.4 64 bit and has been working fine for months. The server is headless and I usually access it through Webmin and Putty.

I have checked on it today and discovered that the server hard drive (files are stored on separate drives) is suddenly full. I have never seen this before and am afraid to delete files without guidance in case I break the server.

The output of df -ah is:

> df -ah
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/wmserver-root
                      227G  227G     0 100% /
proc                     0     0     0   -  /proc
none                     0     0     0   -  /sys
none                     0     0     0   -  /sys/fs/fuse/connections
none                     0     0     0   -  /sys/kernel/debug
none                     0     0     0   -  /sys/kernel/security
none                  491M  220K  491M   1% /dev
none                     0     0     0   -  /dev/pts
none                  496M     0  496M   0% /dev/shm
none                  496M  5.4M  491M   2% /var/run
none                  496M     0  496M   0% /var/lock
none                  496M     0  496M   0% /lib/init/rw
/dev/sdb1             932G  598G  335G  65% /media/backup
/dev/sdc1             932G  303G  630G  33% /media/server
/dev/sda1             228M  190M   26M  89% /boot
/dev/sdd1             932G  598G  335G  65% /mnt/usb_backup

The output from "Disc and Network Fileystems" in Webmin is:




If I edit the mount for / it is mounted as "LVM VG wmserver LV root". On another server I run which is working fine, it is listed as "UUID=32273b44-0e3d-4eb7-a8ac-2cbba4ce1f9f". I am the only one who has access to the server so I'm not sure what has changed.

Can anyone suggest how I can resolve this issue.

Many thanks

[Insomn1a]

http://ubuntuforums.org/showthread.php?t=1502698

[spynappels]

As suggested in the above link, run the following

cd /
du -h --max-depth=1

and see which directory is using up the space. There is a good chance that something has suddenly started writing huge logs in /var/log and that this has filled a disk.

[collisionystm]

I always do

du -chs / | sort -n

[damo12]

Hi everyone,

Thanks for your quick replies. I finally got to the bottom of it with a bit of digging and some questioning of the staff in the office.

One of the mounted drives is an external USB drive (mounted at /mmt). At some point, this drive was accidentally unplugged and then plugged back in at a later date. While the drive was unmounted, the server ran a backup and it created its own /mmt backing up onto the main drive.

When I carried out the df -ah the /mmt looked fine at 65% because the USB drive was re-mounted and working fine.It was only when I unmounted all other drives apart from the "server" drive and carried out df -ah again that I found what had happened.

Ironically, this happened to me years ago and I fixed it with the help of this forum but on that occasion he USB drive was still unmounted so it was easy to see what had happened once I discovered the df -ah command.

Once again, thank you all for your prompt and exact help. Hopefully I won't make this mistake for a third time.

Thanks again.

[redmk2]

Hopefully I won't make this mistake for a third time.

Thanks again.

Here is how I eliminate that problem:

I have the backup script verify that there is no file at the mount point. I create a file called bkp_chk.1st. If this file is found by the backup script -- the partition (drive) is not mounted and the script aborts. If the file is not found then the partition has been mounted and the file can't be seen, so the backup routine is run.

Edit: You could use something like this

if [ -f /mnt/backup/bk_chk.1st ] ; then
  echo "You need to mount the backup drive"
else 
 some code here to run your backup
fi

[damo12]

Here is how I eliminate that problem:

I have the backup script verify that there is no file at the mount point. I create a file called bkp_chk.1st. If this file is found by the backup script -- the partition (drive) is not mounted and the script aborts. If the file is not found then the partition has been mounted and the file can't be seen, so the backup routine is run.

Edit: You could use something like this

if [ -f /mnt/backup/bk_chk.1st ] ; then
  echo "You need to mount the backup drive"
else 
 some code here to run your backup
fi

Hi,

That sounds like a perfect solution but (please excuse my ignorance) where would I store the script and how would I run it? I currently run the backup as a cron job using the command:

rsync -av /media/backup/ /mnt/USB_Backup/ >>/media/cron_log/rsync.lst 2>&1

I'm sorry for asking what I'm sure is a really obvious question but I have no experience of programming or running scripts. I suppose the fact that I have been running Ubuntu servers for the last 6 years and managed so far is a testament to how easy Ubuntu server is to use and how good the forums etc are at providing guidance.

[spynappels]

You would put the command currently in the cronjob (rsync....) in to where the script says "some code here..."

Save the script to some location such as /usr/bin/, remembering to make it executable

sudo chmod +x script_name.sh

In the crontab you just run the script by referencing it with it's full path i.e. /usr/bin/script_name.sh

[damo12]

Thank you so much for your help, I'll try that over the weekend and hopefully will never have this problem again.

[redmk2]

Thank you so much for your help, I'll try that over the weekend and hopefully will never have this problem again.

If you have never created scripts I would start with a small script that is only for testing. You should be able to run it from the terminal. Then you can automate it using cron.

Every sys admin has their own style. For my systems I put locally (to this host) used scripts in /usr/local/sbin. Breaking this down:

usr = unix system resources
local = local to this machine
sbin = system executables (bin is for OS related executables)