Somethings Eating My Drive Space

[Westley]

Hi,

I'm running 16.04.7 LTS terminal only mode and something is now claiming all of my hard drive space on my primary / partition.

This machine is only used as a file server using Samba and, because of Covid, an OpenVPN server.

Yesterday I moved about 120GB from my primary drive to my secondary drive. After the mv was finished, it only showed 60GB of free space. I thought that maybe there was a file system process that still had to complete to mark the rest of the space as free so I didn't worry too much about it.

This morning, that 60GB is not longer free. I just moved another 30GB worth of files and still I'm not showing any free space.

What can I use to try to track down what process is chewing up my hard drive space?

If it makes a difference, I can only connect remotely via SSH at the moment.

Thanks,
Westley

[Westley]

Just for comparison sake, here is the results of du command:
Before moving the 30GB
3.7T .
1.9T ./mnt
1.2T ./srv
589G ./media
86G ./home
2.3G ./usr
2.1G ./var
1.1G ./MySQL
957M ./opt
889M ./lib
589M ./snap
281M ./MongoDB
167M ./etc
133M ./boot
43M ./run
13M ./sbin
13M ./bin
5.5M ./root
88K ./tmp
24K ./BDR
16K ./lost+found
4.0K ./MySQL_old
4.0K ./lib64
0 ./sys
0 ./proc
0 ./dev

After moving the 30GB
3.4T .
1.7T ./mnt
1.1T ./srv
589G ./media
56G ./home
2.3G ./usr
2.1G ./var
1.1G ./MySQL
957M ./opt
889M ./lib
589M ./snap
281M ./MongoDB
167M ./etc
133M ./boot
45M ./run
13M ./sbin
13M ./bin
5.5M ./root
88K ./tmp
24K ./BDR
16K ./lost+found
4.0K ./MySQL_old
4.0K ./lib64
0 ./sys
0 ./proc
0 ./dev


You can see that the /home folder is showing the decrease in size, but still no free space.

Thanks,
Westley

[dinkidonk]

Sounds like the moved files are not being deleted from the source drive. You could try to run this command in order to get the 20 largest files/directories in order to see what gobbles up the most space:

du -abx / | sort -nr | head -n 20

EDIT: You last post suggests that disk space has decreased from 3.7T to 3.4T after moving the 30G and the home folder has decreased from 86G to 56G which is exactly 30G less..

[Westley]

Problem solved!

We had to move the server because of no electricity/internet due to Hurricane Zeta.

The people that moved the server did not move the USB drive that was being used as a secondary backup, but the cron job was still firing every night to copy the files. Since the mount point is still under the / folder, the system just started copy there as if the USB drive was attached.

Thanks for looking.

Westley

[TheFu]

Please mark this thread as solved using the Thread Tools button near the top, so other people don't waste time looking at a solved problem. Please.

ncdu is a handy CLI tool.
A few more commands to help:
This command hides non-storage-impacting mounts:

 $ df -hT -x squashfs -x tmpfs -x devtmpfs

This command shows inode use:

 $ df -iT -x squashfs -x tmpfs -x devtmpfs

 $ cd /var
 $ sudo du -sh *|sort -h

Repeat ...
By sorting the output, we don't need to look at the tiny stuff at the top of the list. Pay attention to what's important.

[Westley]

Hi TheFu,

I may have spoken too early.

First, in regards to ncdu, I couldn't install it since I had no disk space earlier, but not to worry, it is installed now.

One thing I am noticing now is that disk space is disappearing, but since I had over 500G free to work with, it gives me some time.

One culprit seems to be my syslog. In just two hours it has grown to be 368M. In four minutes it grew by 4M, so a meg minute seems to be a bit on the high side. There are a lot of smbd_audit messages, so maybe I need to turn that down.

##
I was at a log level 2 and configured for individual files.

log level = 2
log file = /var/log/samba/log.%m

Changed it to 1. I'll watch it over night.

Thanks,
Westley

[HermanAB]

To avoid the backup to missing removable media problem, I usually make file called NOT-MOUNTED on the mount point and then check that it is not visible, before running a backup and potentially filling up the file system.

[TheFu]

To avoid the backup to missing removable media problem, I usually make file called NOT-MOUNTED on the mount point and then check that it is not visible, before running a backup and potentially filling up the file system.

+1.

I check the mount table.

 
  # Need to verify that the backup partition is mounted
  my $mnt=`/bin/df | grep -c $mnt_point`;
  if ( 0 == $mnt ) {
     print "ERROR: Mount point not mounted: $mnt_point $target_dir \n";
     exit 1;
  }

I use perl for my backup scripts.
Regardless, it is important.

[rbmorse]

To avoid the backup to missing removable media problem, I usually make file called NOT-MOUNTED on the mount point and then check that it is not visible, before running a backup and potentially filling up the file system.

Why I love this place. This became so obvious once I saw it, but it never occurred to me until then.

[dinkidonk]

You cannot mount to a folder with content in it, can you? I am using an inversed method, checking for a file located at the root of the removable media - if not present, do not backup! In fact, the script used to perform the backup with rsync is located at the root of the removable drive!