[SOLVED] Where'd my disk space go?

[fyo]

OK, so I ran out of disk space the other day and started deleting files. After deleting several gigs worth of stuff, "df" still claimed nothing was available (and error messages abounded if I tried creating or copying a file).

Several gigs later, a few hundred megabytes were available.

What the heck is happening to my disk space?

I tried rebooting, I tried everything I could think of and nothing makes any difference. When I do a df, this is what's shown:

/dev/sda2 234G 226G 898M 100% /

Now, my math skills are pretty good, but even if they weren't I would probably still be able to see that something doesn't add up here. 234G total, 226G used, but less than 1 available. OK, so where's the missing disk space?

GParted shows 8GB unused space on sda2.

For fun, I tried resizing sda2, creating a new partition with the unused 8GB. That worked just fine and the new partition did indeed have 8GB free.

Can anyone tell me wtf is going on? This is getting really annoying.

[mattiast]

I had a similar problem in Gutsy, when emptying the trashcan in Nautilus didn't really empying in.. so I had to use the trashcan at the rightbottom on the screen, the bottom panel, then I got all the disk space back again .. very strange problem...

[prshah]

OK, so I ran out of disk space the other day and started deleting files. After deleting several gigs worth of stuff, "df"
Can anyone tell me wtf is going on? This is getting really annoying.

Looks like you're running out of inodes; check the output of

df -i

You get only a limited number of inodes (usually 1%) which store information about files and directories. Having lots of tiny files and/or heavily nested directories will eat up inodes.

I believe you can change the default inode allocation; but I've never found the need for it and can't really tell you how.

[Wandering]

Free space is always significantly less than disk space minus used space because of a number of bookeeping issues and sparse storage. There is an equivalent to checkdisk in Ubuntu, and it would be good to run it. It should run automatically every 37 boots unless you disabled it.

Remember, deleting a small file only leaves a small place to put another file, and many small files scattered across the disk when deleted free a large space, but not all in one piece, and so with limited usefulness.

Good luck.

[unutbu]

Please post

sudo tune2fs -l /dev/sda2

From the tune2fs man page:

-m reserved-blocks-percentage
   Set the percentage of the filesystem which may only be allocated by priv‐
   ileged processes.   Reserving some number of filesystem blocks for use by
   privileged processes is done to avoid filesystem  fragmentation,  and  to
   allow  system  daemons,  such as syslogd(8), to continue to function cor‐
   rectly after non-privileged processes are prevented from writing  to  the
   filesystem.  Normally, the default percentage of reserved blocks is 5%.

I think it is a little silly that the percentage is set at 5% since 5% could be measly on a small disk and ridiculously large on a huge disk. On your 234GB disk, 5% would be almost 12GB for stuff like syslogd. (Run `du -sh /var/log` to see how big /var/log is. syslogd writes there.) I'm no expert, but 12GB seems excessive to me.

If you want to reduce the amount of reserved space to say 1%, you can run

sudo tune2fs -m1 /dev/sda2

[fyo]

Please post

sudo tune2fs -l /dev/sda2

$ sudo tune2fs -l /dev/sda2
tune2fs 1.40.8 (13-Mar-2008)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          4c924613-ac6b-4dc7-af2e-2a8734f0f820
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash 
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              31145984
Block count:              62291968
Reserved block count:     1868757
Free blocks:              4753315
Free inodes:              30735049
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1009
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16384
Inode blocks per group:   512
Filesystem created:       Thu Aug  2 13:13:43 2007
Last mount time:          Thu May 15 14:59:37 2008
Last write time:          Thu May 15 14:59:37 2008
Mount count:              2
Maximum mount count:      36
Last checked:             Thu May 15 14:24:02 2008
Check interval:           15552000 (6 months)
Next check after:         Tue Nov 11 13:24:02 2008
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:		  128
Journal inode:            8
First orphan inode:       26689673
Default directory hash:   tea
Directory Hash Seed:      b987c31f-5baa-43ed-86df-f74b5e615d51
Journal backup:           inode blocks

1.8M reserved blocks x 4k block size = 7GB reserved space.

Hmm... this could go some ways to explaining the initial issue.

When I originally ran out of disk space, I was transferring some video from a DV camera over firewire. Now, by default in Ubuntu, accessing firewire requires the use of sudo (yes, I realize that adding a dedicated user for the device would probably be the right way to do it, but I was lazy and did it the default Ubuntu way), so I probably gobbled this reserved space right up with my root process...

Still... I'm MISSING quite a chunk of disk space since before this whole debacle. As reported in my previous post, the total size of sda2 as reported by df has dropped by 13GB despited having been "fed" 4GB from my swap partition.

Worryingly, the total space reported by df matches the total space calculated from the total block count in the output above (234GB vs 237GB).

GParted currently reports sda2 as beign 254.79GiB (with 18.13GiB unused). "df" claims 234GB total size and 12GB available (yes, I freed up a bunch of more space). This is weird on a number of levels:

- Why is the total size as reported by df so much lower than that reported by GParted?
- Why is df unable to do math? It reports the following when NOT specifying -h (human readable):

total size: 245258456
used: 226245676
available: 11537752

And the following when -h is specified:

total size: 234GB
used: 216GB
available: 12GB

Now, the first two sizes (total and used) are correctly calculated (with proper rounding, even). But the "available" space is off by a gigabyte (it should be 11.00GB, which cannot be rounded to 12GB).

Still, my main concern now is the 13GB or so that I'm just plain missing since before this whole mess started. Space that's simply disappeared from the size reported by df, despite having increased the size of the partition.

Any chance GParted f*cked (fscked?) by disk?

[unutbu]

GParted measures the size of the partition,
`df` measures the size of the filesystem that has been formated onto the partition.
A file system needs to record superblocks which store the structure of the filesystem. Ext3 is also a journaled filesystem and the journaling takes up space too.

GParted reports sda2 as 254.79 GiB = 254.79 * 10^9 / 2^30 = 237.3 GB.
`df` reports 234GB. The difference is 3.3GB. This, apparently, is the amount of space ext3 reserves to manage the filesystem structure.

`df` reports
total=245258456 1K-blocks.
used=226245676 1K-blocks.

total-used = 19012780 1K-blocks.

`df` reports available=11537752 1K-blocks.

The discrepancy = (total-used)-available = 19012780-11537752 = 7475028 1K-blocks.

tune2fs reports reserved blocks = 1868757 4K-blocks. That equals 1868757*4 = 7475028 1K-blocks.

So the "discrepancy" is really just the reserved blocks. `df` output is unintuitive, but correct when properly interpreted.

Now putting all this aside, you say `df -h` reported sda2 started out with 247GB. I have no idea how that could have changed to 234GB.

I hope the above explanation, however, makes you a bit more comfortable believing the output reported by both gparted and df. If so, then you can look at the size of each of your partitions as reported by gparted, add that up and see if it agrees with the reported size of your hard drive. It you get agreement then at least, in some sense, you'll know you're not missing 13GB.

[fyo]

inodes: I'm using 411k of 31M inodes.

checkdisk: I already ran "e2fsck -f -v" on it and no problems were detected.

bookkeeping and sparse storage: Last one first... "sparse" storage refers to files only taking up what they actually contain and not the fully allocated amount. This is not supported on all file systems, but is on ext3, which is what I use.

I do have a bunch of "sparse" files and I certainly realize that if I delete a sparse file which claims to contain, say, 1GB, only whatever space is actually allocated to that file will be released.

None of the files I deleted to get more space were sparse files and I don't see how the issue would affect the output of "df" (total-avail >> free).

Note that I'm also well aware of the "du" command and the total usage reported by it matches that of "df" quite well.

For the hell of it, I tried giving the ubuntu partition half my swap partition. The result was 400MB more free disk space on the ubuntu partition after getting half my 8GB swap partition. Looks like I just lost another 4GB to the beast...

Anyhooo.... I just noticed another thing. I previously saved the output of df and noticed a strange discrepancy. The total size of sda2 (the ubuntu partition) used to be reported as 247GB. After feeding it MORE space from other partitions, the total size is now reported as 234GB.

WTF?