pyfragtools: Defragmenter for Linux

[jdong]

Hey, guess who got bored last weekend on a robotics trip and decided to write a Linux defragger?

A quote from the README:

defrag
==========

INTRO:

    Defrag is a set of scripts designed to measure and cope with filesystem fragmentation. Currently, it only consists of "defrag".
    
USAGE:
    
    Fragtool's syntax is:
    
    /usr/sbin/defrag [-h] [-n passes] [--passes n] [-t threshold] [--threshold n] [-a] [--analyze] [--help] path
    
    When invoked, defrag will analyze the given path recursively for file fragmentation and print out a summary of its findings. Next, it will ask you if you'd like to run any defragmentation passes.
    Fragtool also takes an optional passes argument, which makes it perform fully non-interactively. Positive number of passes will perform that many passes and exit, while zero passes will just print out the summary and exit.
    
==========
FAQ
==========

Q: A defragger? For Linux? Are you crazy?
A: No, I certainly am not. Under certain circumstances, even the most fragmentation-resistant filesystems get fragmented. Don't believe me? Look around with defrag. Whether or not filesystem performance is affected can be debatable at times, though I personally have to say that when a 300MB torrent is split into 5000 fragments, read speed is drastically affected.

Q: What filesystems is it compatible with?
A: Any filesystem that Linux can write to. The tool is pretty filesystem agnostic, but does require that filefrags be able to understand the layout of files.

Q: Is it safe to use?
A: Defrag works great in the author's extensive testing with all kinds of files. However, I cannot be certain that it does not damage your setup, so use with caution like any other fragmentation tool. As long as you use a journaling filesystem, the process is atomic. Before defragging a file, we make sure that it is not open for write access. Also, before doing the in-place replacement at the end of a defrag, we make sure again that the file has not changed. These are the same precautions that the tried-and-true xfs_fsr XFS defragger uses.

Q: Is it safe to interrupt?
A: Yes, defrag will safely clean up after itself. In the case of abrupt termination (SIGKILL, power off, etc), fragtools will leave behind ".defrag" temporary folders on the filesystem. They are owned by root with a mode of 000, and can be easily located and cleaned up using "find".

Q: Does it resume where it left off after an interrupt?
A: No. It doesn't. It sounds cool though.

Q: Is it effective?
A: Depends on the particular filesystem it is being used on. The defragmentation algorithm either fully defragments a file on the first try, get progressively better in subsequent passes, or is unable to make any improvement for infinite numbers of passes. Usually, if the file is not defragmented by the 10th pass, it's a lost cause. But in general, you will see improvements on fragmented filesystems. You will not see great improvements if you obsessive-compulsively run the defragger on the same set of files. Get a life.

Q: How many passes should I run?
A: Generally, anything from 5-10 passes is good. If it's not defragmented by then, chances are that your filesystem does not have enough contiguous free space to defragment the file. Go by what you feel is good after running the 5-10 passes.

Q: I run XFS. Should I use this, or xfs_fsr?
A: If you are interested in defragmenting, use xfs_fsr. It is faster by orders of magnitude, not to mention that it is production-tested. If you would like to get fragmentation statistics, defrag is the perfect tool to get that.

Q: If you think this is so great, why hasn't anyone else written a defragmenter for Linux?
A: Others indeed have. A user at forums.gentoo.org wrote a perl fragmentation checker, and Con Kolivas wrote a bash script defragger. Fragtool was inspired by both of these efforts, and aims to intelligently combine the two to perform the task more intelligently (i.e. do not try defragmenting defragmented files, do not make fragmentation worse than before, etc).

Sample output:

jdong@jdong-laptop:~$ sudo defrag /usr/sbin
Building list of files to analyze... done!

Analyze finished. 1.0 % fragmentation (2 files), 13.3 average frags/MB
Fragmented files:
13.92   /usr/sbin/sshd
12.74   /usr/sbin/gdm
How many passes to run? [10]

111111111111111111111111111111111111111111111111111111111111
===> Pass 1 of 10 <===    Remaining Fragmentation 26/26 (100%)
111111111111111111111111111111111111111111111111111111111111


  Pass 1 of 10, 1/2 (50%): 13.9 frags/MB /usr/sbin/sshd
           301372 100%   23.29MB/s    0:00:00  (1, 100.0% of 1)                      Fully defragmented!

  Pass 1 of 10, 2/2 (100%): 12.7 frags/MB /usr/sbin/gdm
           246864 100%   51.04MB/s    0:00:00  (1, 100.0% of 1)                      Improved: (12.7 --> 8.5)

222222222222222222222222222222222222222222222222222222222222
===> Pass 2 of 10 <===    Remaining Fragmentation 8/26 (31%)
222222222222222222222222222222222222222222222222222222222222


  Pass 2 of 10, 1/1 (100%): 8.5 frags/MB /usr/sbin/gdm
           246864 100%   68.06MB/s    0:00:00  (1, 100.0% of 1)                      Fully defragmented!

333333333333333333333333333333333333333333333333333333333333
===> Pass 3 of 10 <===    Remaining Fragmentation 0/26 (0%)
333333333333333333333333333333333333333333333333333333333333


444444444444444444444444444444444444444444444444444444444444
===> Pass 4 of 10 <===    Remaining Fragmentation 0/26 (0%)
444444444444444444444444444444444444444444444444444444444444


555555555555555555555555555555555555555555555555555555555555
===> Pass 5 of 10 <===    Remaining Fragmentation 0/26 (0%)
555555555555555555555555555555555555555555555555555555555555


666666666666666666666666666666666666666666666666666666666666
===> Pass 6 of 10 <===    Remaining Fragmentation 0/26 (0%)
666666666666666666666666666666666666666666666666666666666666


777777777777777777777777777777777777777777777777777777777777
===> Pass 7 of 10 <===    Remaining Fragmentation 0/26 (0%)
777777777777777777777777777777777777777777777777777777777777


888888888888888888888888888888888888888888888888888888888888
===> Pass 8 of 10 <===    Remaining Fragmentation 0/26 (0%)
888888888888888888888888888888888888888888888888888888888888


999999999999999999999999999999999999999999999999999999999999
===> Pass 9 of 10 <===    Remaining Fragmentation 0/26 (0%)
999999999999999999999999999999999999999999999999999999999999


101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010
===> Pass 10 of 10 <===    Remaining Fragmentation 0/26 (0%)
101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010

========= COMPLETE ===========
Remaining Fragmented Files:
Frags/MB Before:         26.66
Frags/MB After:          0.00
Improvement:             100.0 %
===============================
jdong@jdong-laptop:~$

Attached is a tarball containing both debianized source code (written in Python) and a .deb package for simpler installation. There is only one file (defrag) that needs to be installed. It should be very safe to use (follows the trusted XFS defragger algorithm) on any filesystem, but I don't guarantee that it'll be flawless on every setup. Don't run passes over crucial data unless you have backups, I guess


The fragmentation numbers printed are in Frags/MB. I think that makes more sense than just counting fragments, so you don't spend time defragging huge files that will naturally be slightly fragmented.


Enjoy!


Small Note
Do _not_ obsessive-compulsively run passes trying to achieve perfection. By default, the defragger counts files with less than .5frags/MB as unfragmented. This can be overridden with the -t parameter. Do it wisely... If you try too hard to defrag, what will happen is that while files may defragment, free space will become fragmented in the process, which will ultimately lower write performance.

This is not a full defragmenter (i.e. free space consolidation, directory reorganization), but just a file defragger.



The source code is under the GPL, and you are welcome to play around with it. I'm open to suggestions and contributions. The working directory of the source archive is already a bzr 0.8 knits repository, so it's ready for decentralized commits.

I will be publishing the head branch as soon as I find a place to host it, as my system will not take the bandwidth hit and Launchpad supermirror does not support Knits yet.

[TheCaptain]

O&O has had a defragmenter for Linux for years, i tried it on servers, the database efficiancy went up by... 0%, not measurable.

You can try this out if you have a recent backup and a lot of time on your hands but the real reason one is not provided in ANY distro is that it's not needed, the improvements aren't measurable.

[htinn]

Hmm. Anything I need to install first?

ImportError: No module named bzrlib.osutils

[jdong]

Hmm. Anything I need to install first?

ImportError: No module named bzrlib.osutils

You may need to install bzr. I will try to remove that dependency... The progress bar was stolen from bzrlib

[ubongo2008]

O&O has had a defragmenter for Linux for years, i tried it on servers, the database efficiancy went up by... 0%, not measurable.

You can try this out if you have a recent backup and a lot of time on your hands but the real reason one is not provided in ANY distro is that it's not needed, the improvements aren't measurable.

IMHO ext3, ext4 doesn't need something like an defragmenter because of the filesystem it just doesn't fragment files. maybe in some cases it would be good to pack files near together on the drive but i think ext does not fragment files (nor do SSD's by design)

well at least that is what i learned some time ago in my class

Edit: concerning the time ... i think an backup and restore of the partition will be faster with same benefits if not done sector by sector

EDIT: ok there seems to be some fragmentation too but also seems to be minimal ... no comparison to windows ntfs at all

[jdong]

IMHO ext3, ext4 doesn't need something like an defragmenter because of the filesystem it just doesn't fragment files. maybe in some cases it would be good to pack files near together on the drive but i think ext does not fragment files (nor do SSD's by design)

well at least that is what i learned some time ago in my class

Edit: concerning the time ... i think an backup and restore of the partition will be faster with same benefits if not done sector by sector

EDIT: ok there seems to be some fragmentation too but also seems to be minimal ... no comparison to windows ntfs at all

This is a long long thread, but compelling cases have been made in the first few pages of when ext3 and ext4 indeed DO get fragmented (try downloading a 10GB torrent at 10KB/s without preallocation -- the file will be so fragmented that it'll read back at much less than 10% peak speed). The documentation warns users that in most cases, defragmentation is not necessary.

[TheCaptain]

Oh, and XFS is in no way safe to run on PC hardware without a good UPS that will gently shut down your system, that goes for most other filesystems like it.

[jdong]

Well, on some mythtv setups, I have seen 100MB files with 7000+ fragments (ext3 & reiser), and at that point read performance degrades to the point that it can't keep up with burning to a 48x CD (approx 5MB/s, on a 40MB/s drive). A simple copy turned it down to 50 or so fragments, and read performance drastically improved.

That's why there's the built-in limit of 0.5frags/MB, which IMO should be even higher than that.

As far as system performance, unless you do something special that fragments up the disk (torrenting, mythtv, large downloads), you probably won't realize much, but it's always good to know you're ready for something like this, or if you're curious about fragmentation the analyze mode is great.

[TheCaptain]

Well, on some mythtv setups, I have seen 100MB files with 7000+ fragments (ext3 & reiser), and at that point read performance degrades to the point that it can't keep up with burning to a 48x CD (approx 5MB/s, on a 40MB/s drive). A simple copy turned it down to 50 or so fragments, and read performance drastically improved.

That's why there's the built-in limit of 0.5frags/MB, which IMO should be even higher than that.

As far as system performance, unless you do something special that fragments up the disk (torrenting, mythtv, large downloads), you probably won't realize much, but it's always good to know you're ready for something like this, or if you're curious about fragmentation the analyze mode is great.

You're saying that simultaneous read/write from multiple users on a database server will not fragment like torrenting, mythtv, large downloads?

We had extreme fragmentation, the real world results of an O&O defragment were not measurable, thus we opted not to implement the function in the nightly runs.

Provide me with real world benchmarks and i'll trust that it will improve anything, for now it's just another way to potentially screw up my system with no real benefit to be had.

[TheCaptain]

If you are curious about fragmentation run fsck, if you are worried that it may affect your performance, install more memory, afaik the cache subsystem of Linux is second to none and VERY able to keep up with the demand of burning a cd or dvd, i'd say that burning a cd has more to do with the interface than the ability to deliver when it comes to write speed.

In 110% of the cases when a burn has to slow down it's because of the writer, the media or the interface, it's never ever because the HDD cannot read (while fully cached) as fast as the burner can write.

It's not generally known but a fast read doesn't flow to a fast write, both adds to the PCI bus by requests and usually blams the inferface with crap, this happens regardless of fragmentation and is why the QOS of SCSI interfaces is so much smoother, even though it has to be translated.