Does linux Need Defrag?

[nakama85]

Thanks Everyone! So Far Paddy Landau's solution seemed to help a bit.
I am thinking though, to get everything back to normal I will need to reinstall.
Thanks again everyone for your help!

[admelfo]

Starting to think that my old Pentium III Coppermine processor just can't handle it anymore.

[fluxlizard]

Starting to think that my old Pentium III Coppermine processor just can't handle it anymore.

try pclinuxos gnome edition for p3 processor. I have that for my old p3 700 mhz laptop with 256 mb ram and it runs just fine...

[irne.barnard]

I'd actually strongly suggest not defraging ... the reason behind this? Even on windows most defraggers have 2 options, 1 Defragment, 2 Compact ... sometimes called something different, but the end result's the same.

The "defragment" is supposed to make all files into contiguous blocks. The "compact" is supposed to defragment and make the free space into a contiguous block. Now while this sounds nice, the reality is quite different:

Fragmentation does cause productivity loss as each file access now could become hundreds of HDD reads / writes. The file system makes all the difference:

1. FAT based file systems save each file directly following each other. So if you later edit / add to this file, the added portion needs to be saved somewhere else. This will create a fragment (or more than one). A new file is saved starting from the 1st blank spot, even if that blank spot is too small to contain the entire file.
2. NTFS is a bit better in theory, it allows some free space around each file. Then if it notices that the file will become fragmented, it "attempts" to save the entire file in a new location. The caveat to this is "efficiency": if it would not take too much time to save the entire file over again it will be done, otherwise just create a new fragment. Just how it determines "too much time" is up for grabs ... and like most M$ ideas also a secret!
3. Ext3/4 also generates blank spaces "around" each file. Where it differs significantly from NTFS is that it'll only fragment when it's impossible to keep the file as one single fragment ... that (usually) only happens when the disk becomes too full.

Why defrag utilities don't work:

Defragment:
All this really does is what ext3/4 already does ... but after the fact. It checks all the files on the HDD, then for all those with more than one fragment it "tries" to move them to an empty space which is large enough. After a while you end up with lots and lots of small bits and pieces of empty space throughout the HDD. With Linux's ext3/4 these are actually designed to prevent fragmentation, they're given a size by ratio and each time the file grows beyond that space it's moved somewhere else and given a larger "buffer zone". With a windows defrag they happen per accident and the size can be anything from one allocation unit (usually 4096 bytes) to may gigs. There's no real "plan" behind it, and it doesn't matter how many times the file's grown or what its size is. It's even possible to have absolutely no space between files, so you end up with a FAT like scenario.

Compact:
This is the worst thing you could do to your HDD. Apart from the fact that you're diminishing the mechanical life by overworking it, compacting will cause fragmentation in the future. This is because it literally rearranges files to match that found on FAT systems. So the very first edit after a compact will create extra fragments (at least one extra). Some defragers (like Norton Speed Disc) allows for "smart" rearrangement of files. All they basically do is move files which are changed a lot (like documents) to the end of the disk, and all non-changing files (like programs) to the start. They still remove the buffer zones between files. The only time I'd even consider something like this is when the HDD will only be used for read access, but even then there's sometimes better arrangements performance wise.

All that said ... absolutely nothing is ever perfect. Even ext4 systems do become fragmented (but very seldom such that performance is lost). Main culprits of causing fragmentation on ext3/4 are database systems. These have multiple read & writes in concurrent environments, thus rearrangement to remove fragmentation could actually damage performance. In such cases it's usually a good idea to "now and again" copy the fragmented file off to an external HDD, delete the original & copy it back. This method lets the ext file system "decide for itself" where to best place the previously fragmented file. And then of course don't over fill your HDD, a good limit is around 50%. But absolutely never fill it above 80-90%, that's just asking for trouble (no matter what OS / File System you have)!

Some DBMS's have ways of getting around this. E.g. Interbase (or Firebird) makes one file for its database which has exactly a specified size, if the data grows beyond this filesize a 2nd new file is created instead of just growing the 1st. This ensures that each file is only one fragment - but obviously the file may contain quite a lot of wasted space. This is not the default, but for serious DB's it's recommended.

[Paddy Landau]

I'd actually strongly suggest not defraging...

Thank you for a clear explanation. I'll certainly keep these ideas in mind next time I work on Windows!

[ndefontenay]

Excellent explanation of what happens to files on a disk.

Being a Database Administrator, I have to plus what has been said above. Databases are big fragment generators... IF you don't do a little planning before hand.

If anyone is planning on creating a production database be it for a home business or something bigger, don't let the database manage its data file growth. Try to plan ahead by allocating the amount of disk space you think the related database will take in the coming 2 years. This will create a big empty file and the DB will manage its data inside the empty file. This will create less fragmentation in the long run.

This is a very summarize explanation. Best practices are a bit more extensive than that but this is the idea.

The worst case is usually the default scenario. A database is created with a default tablespace of say 100 MB then it increases by 2MB when full. This is a highway to hell.

[irne.barnard]

Thanks, I did my own googling about this. Seeing as I've been on the dark side since the DOS days (only recently coming to the light ) ... I've also been missing Defrag. Yet even in DOS (FAT) I noticed the utter uselessness of a full Compact Defragmentation, as described in my previous post.

Ext file system is not absolutely perfect as yet. I'd like to see something which "learns" which files usually grow by how much, so that the more you use the system the larger those files' buffer zones become (and the smaller those which don't change).

@ndefontenay: What DBMS do you generally use? As per my explanation I prefer Firebird as much of that pre-planning becomes automatic, but that's only one of the reasons.

[lasleym]

Paddy - this seemingly tiny little fix did noticeable wonders to my FF. Our system is about 26 days old, so just about the month you suggested.

Thanks for the 'speedy' fix.

I've found that Firefox (and Thunderbird) can lose speed over time as their databases become fragmented or whatever.

It's easy to fix. I forget where I found this, but it sped up my Firefox considerably.

First, install sqlite3 if not already installed (you can just click that link, or use System > Administration > Synaptic Package Manager).

Then, on a regular basis (say, once a month):

1. Close all instances of Firefox, Thunderbird, Seamonkey. Possibly Camino, but I don't know about that.
2. Just in case, double-check:
   

   sudo killall firefox thunderbird thunderbird-bin seamonkey

3. Reorganise the Mozilla databases:
   

   sudo find /home -xdev \( -name '*.sqlite' \) -exec sqlite3 {} 'vacuum' \;

   This command finds every file *.sqlite in all /home subfolders and runs sqlite3 vacuum on each. It takes a few minutes to run, so be patient.

HTH

[Ozitraveller]

FYI Ubuntu will defrag itself every 25 (I think, maybe 30) starts, ans fragmentation is usually around 3 - 5%

[alphaniner]

FYI Ubuntu will defrag itself every 25 (I think, maybe 30) starts, ans fragmentation is usually around 3 - 5%

That's the automatic file system check (fsck), not a defrag. It does report the fragmentation though.