Ubuntu Forums - View Single Post

irne.barnard · 05:50 AM

I'd actually strongly suggest not defraging ... the reason behind this? Even on windows most defraggers have 2 options, 1 Defragment, 2 Compact ... sometimes called something different, but the end result's the same.

The "defragment" is supposed to make all files into contiguous blocks. The "compact" is supposed to defragment and make the free space into a contiguous block. Now while this sounds nice, the reality is quite different:

Fragmentation does cause productivity loss as each file access now could become hundreds of HDD reads / writes. The file system makes all the difference:

FAT based file systems save each file directly following each other. So if you later edit / add to this file, the added portion needs to be saved somewhere else. This will create a fragment (or more than one). A new file is saved starting from the 1st blank spot, even if that blank spot is too small to contain the entire file.
NTFS is a bit better in theory, it allows some free space around each file. Then if it notices that the file will become fragmented, it "attempts" to save the entire file in a new location. The caveat to this is "efficiency": if it would not take too much time to save the entire file over again it will be done, otherwise just create a new fragment. Just how it determines "too much time" is up for grabs ... and like most M$ ideas also a secret!
Ext3/4 also generates blank spaces "around" each file. Where it differs significantly from NTFS is that it'll only fragment when it's impossible to keep the file as one single fragment ... that (usually) only happens when the disk becomes too full.

Why defrag utilities don't work:

Defragment:
All this really does is what ext3/4 already does ... but after the fact. It checks all the files on the HDD, then for all those with more than one fragment it "tries" to move them to an empty space which is large enough. After a while you end up with lots and lots of small bits and pieces of empty space throughout the HDD. With Linux's ext3/4 these are actually designed to prevent fragmentation, they're given a size by ratio and each time the file grows beyond that space it's moved somewhere else and given a larger "buffer zone". With a windows defrag they happen per accident and the size can be anything from one allocation unit (usually 4096 bytes) to may gigs. There's no real "plan" behind it, and it doesn't matter how many times the file's grown or what its size is. It's even possible to have absolutely no space between files, so you end up with a FAT like scenario.

Compact:
This is the worst thing you could do to your HDD. Apart from the fact that you're diminishing the mechanical life by overworking it, compacting will cause fragmentation in the future. This is because it literally rearranges files to match that found on FAT systems. So the very first edit after a compact will create extra fragments (at least one extra). Some defragers (like Norton Speed Disc) allows for "smart" rearrangement of files. All they basically do is move files which are changed a lot (like documents) to the end of the disk, and all non-changing files (like programs) to the start. They still remove the buffer zones between files. The only time I'd even consider something like this is when the HDD will only be used for read access, but even then there's sometimes better arrangements performance wise.

All that said ... absolutely nothing is ever perfect.

Even ext4 systems do become fragmented (but very seldom such that performance is lost). Main culprits of causing fragmentation on ext3/4 are database systems. These have multiple read & writes in concurrent environments, thus rearrangement to remove fragmentation could actually damage performance. In such cases it's usually a good idea to "now and again" copy the fragmented file off to an external HDD, delete the original & copy it back. This method lets the ext file system "decide for itself" where to best place the previously fragmented file. And then of course don't over fill your HDD, a good limit is around 50%. But absolutely never fill it above 80-90%, that's just asking for trouble (no matter what OS / File System you have)!

Some DBMS's have ways of getting around this. E.g. Interbase (or Firebird) makes one file for its database which has exactly a specified size, if the data grows beyond this filesize a 2nd new file is created instead of just growing the 1st. This ensures that each file is only one fragment - but obviously the file may contain quite a lot of wasted space. This is not the default, but for serious DB's it's recommended.

October 26th, 2009	#14
irne.barnard Spilled the Beans Join Date: Sep 2009 Location: Jo'burg SA Beans: 13 Ubuntu Studio 8.04 Hardy Heron	Re: Does linux Need Defrag? I'd actually strongly suggest not defraging ... the reason behind this? Even on windows most defraggers have 2 options, 1 Defragment, 2 Compact ... sometimes called something different, but the end result's the same. The "defragment" is supposed to make all files into contiguous blocks. The "compact" is supposed to defragment and make the free space into a contiguous block. Now while this sounds nice, the reality is quite different: Fragmentation does cause productivity loss as each file access now could become hundreds of HDD reads / writes. The file system makes all the difference: FAT based file systems save each file directly following each other. So if you later edit / add to this file, the added portion needs to be saved somewhere else. This will create a fragment (or more than one). A new file is saved starting from the 1st blank spot, even if that blank spot is too small to contain the entire file. NTFS is a bit better in theory, it allows some free space around each file. Then if it notices that the file will become fragmented, it "attempts" to save the entire file in a new location. The caveat to this is "efficiency": if it would not take too much time to save the entire file over again it will be done, otherwise just create a new fragment. Just how it determines "too much time" is up for grabs ... and like most M$ ideas also a secret! Ext3/4 also generates blank spaces "around" each file. Where it differs significantly from NTFS is that it'll only fragment when it's impossible to keep the file as one single fragment ... that (usually) only happens when the disk becomes too full. Why defrag utilities don't work: Defragment: All this really does is what ext3/4 already does ... but after the fact. It checks all the files on the HDD, then for all those with more than one fragment it "tries" to move them to an empty space which is large enough. After a while you end up with lots and lots of small bits and pieces of empty space throughout the HDD. With Linux's ext3/4 these are actually designed to prevent fragmentation, they're given a size by ratio and each time the file grows beyond that space it's moved somewhere else and given a larger "buffer zone". With a windows defrag they happen per accident and the size can be anything from one allocation unit (usually 4096 bytes) to may gigs. There's no real "plan" behind it, and it doesn't matter how many times the file's grown or what its size is. It's even possible to have absolutely no space between files, so you end up with a FAT like scenario. Compact: This is the worst thing you could do to your HDD. Apart from the fact that you're diminishing the mechanical life by overworking it, compacting will cause fragmentation in the future. This is because it literally rearranges files to match that found on FAT systems. So the very first edit after a compact will create extra fragments (at least one extra). Some defragers (like Norton Speed Disc) allows for "smart" rearrangement of files. All they basically do is move files which are changed a lot (like documents) to the end of the disk, and all non-changing files (like programs) to the start. They still remove the buffer zones between files. The only time I'd even consider something like this is when the HDD will only be used for read access, but even then there's sometimes better arrangements performance wise. All that said ... absolutely nothing is ever perfect. Even ext4 systems do become fragmented (but very seldom such that performance is lost). Main culprits of causing fragmentation on ext3/4 are database systems. These have multiple read & writes in concurrent environments, thus rearrangement to remove fragmentation could actually damage performance. In such cases it's usually a good idea to "now and again" copy the fragmented file off to an external HDD, delete the original & copy it back. This method lets the ext file system "decide for itself" where to best place the previously fragmented file. And then of course don't over fill your HDD, a good limit is around 50%. But absolutely never fill it above 80-90%, that's just asking for trouble (no matter what OS / File System you have)! Some DBMS's have ways of getting around this. E.g. Interbase (or Firebird) makes one file for its database which has exactly a specified size, if the data grows beyond this filesize a 2nd new file is created instead of just growing the 1st. This ensures that each file is only one fragment - but obviously the file may contain quite a lot of wasted space. This is not the default, but for serious DB's it's recommended. Last edited by irne.barnard; October 26th, 2009 at 05:50 AM..