Page 1 of 31 12311 ... LastLast
Results 1 to 10 of 309

Thread: pyfragtools: Defragmenter for Linux

  1. #1
    Join Date
    Oct 2004
    Location
    Cupertino, CA
    Beans
    5,092
    Distro
    Ubuntu 10.04 Lucid Lynx

    pyfragtools: Defragmenter for Linux

    Hey, guess who got bored last weekend on a robotics trip and decided to write a Linux defragger?

    A quote from the README:
    Code:
    defrag
    ==========
    
    INTRO:
    
        Defrag is a set of scripts designed to measure and cope with filesystem fragmentation. Currently, it only consists of "defrag".
        
    USAGE:
        
        Fragtool's syntax is:
        
        /usr/sbin/defrag [-h] [-n passes] [--passes n] [-t threshold] [--threshold n] [-a] [--analyze] [--help] path
        
        When invoked, defrag will analyze the given path recursively for file fragmentation and print out a summary of its findings. Next, it will ask you if you'd like to run any defragmentation passes.
        Fragtool also takes an optional passes argument, which makes it perform fully non-interactively. Positive number of passes will perform that many passes and exit, while zero passes will just print out the summary and exit.
        
    ==========
    FAQ
    ==========
    
    Q: A defragger? For Linux? Are you crazy?
    A: No, I certainly am not. Under certain circumstances, even the most fragmentation-resistant filesystems get fragmented. Don't believe me? Look around with defrag. Whether or not filesystem performance is affected can be debatable at times, though I personally have to say that when a 300MB torrent is split into 5000 fragments, read speed is drastically affected.
    
    Q: What filesystems is it compatible with?
    A: Any filesystem that Linux can write to. The tool is pretty filesystem agnostic, but does require that filefrags be able to understand the layout of files.
    
    Q: Is it safe to use?
    A: Defrag works great in the author's extensive testing with all kinds of files. However, I cannot be certain that it does not damage your setup, so use with caution like any other fragmentation tool. As long as you use a journaling filesystem, the process is atomic. Before defragging a file, we make sure that it is not open for write access. Also, before doing the in-place replacement at the end of a defrag, we make sure again that the file has not changed. These are the same precautions that the tried-and-true xfs_fsr XFS defragger uses.
    
    Q: Is it safe to interrupt?
    A: Yes, defrag will safely clean up after itself. In the case of abrupt termination (SIGKILL, power off, etc), fragtools will leave behind ".defrag" temporary folders on the filesystem. They are owned by root with a mode of 000, and can be easily located and cleaned up using "find".
    
    Q: Does it resume where it left off after an interrupt?
    A: No. It doesn't. It sounds cool though.
    
    Q: Is it effective?
    A: Depends on the particular filesystem it is being used on. The defragmentation algorithm either fully defragments a file on the first try, get progressively better in subsequent passes, or is unable to make any improvement for infinite numbers of passes. Usually, if the file is not defragmented by the 10th pass, it's a lost cause. But in general, you will see improvements on fragmented filesystems. You will not see great improvements if you obsessive-compulsively run the defragger on the same set of files. Get a life.
    
    Q: How many passes should I run?
    A: Generally, anything from 5-10 passes is good. If it's not defragmented by then, chances are that your filesystem does not have enough contiguous free space to defragment the file. Go by what you feel is good after running the 5-10 passes.
    
    Q: I run XFS. Should I use this, or xfs_fsr?
    A: If you are interested in defragmenting, use xfs_fsr. It is faster by orders of magnitude, not to mention that it is production-tested. If you would like to get fragmentation statistics, defrag is the perfect tool to get that.
    
    Q: If you think this is so great, why hasn't anyone else written a defragmenter for Linux?
    A: Others indeed have. A user at forums.gentoo.org wrote a perl fragmentation checker, and Con Kolivas wrote a bash script defragger. Fragtool was inspired by both of these efforts, and aims to intelligently combine the two to perform the task more intelligently (i.e. do not try defragmenting defragmented files, do not make fragmentation worse than before, etc).


    Sample output:

    Code:
    jdong@jdong-laptop:~$ sudo defrag /usr/sbin
    Building list of files to analyze... done!
    
    Analyze finished. 1.0 % fragmentation (2 files), 13.3 average frags/MB
    Fragmented files:
    13.92   /usr/sbin/sshd
    12.74   /usr/sbin/gdm
    How many passes to run? [10]
    
    111111111111111111111111111111111111111111111111111111111111
    ===> Pass 1 of 10 <===    Remaining Fragmentation 26/26 (100%)
    111111111111111111111111111111111111111111111111111111111111
    
    
      Pass 1 of 10, 1/2 (50%): 13.9 frags/MB /usr/sbin/sshd
               301372 100%   23.29MB/s    0:00:00  (1, 100.0% of 1)                      Fully defragmented!
    
      Pass 1 of 10, 2/2 (100%): 12.7 frags/MB /usr/sbin/gdm
               246864 100%   51.04MB/s    0:00:00  (1, 100.0% of 1)                      Improved: (12.7 --> 8.5)
    
    222222222222222222222222222222222222222222222222222222222222
    ===> Pass 2 of 10 <===    Remaining Fragmentation 8/26 (31%)
    222222222222222222222222222222222222222222222222222222222222
    
    
      Pass 2 of 10, 1/1 (100%): 8.5 frags/MB /usr/sbin/gdm
               246864 100%   68.06MB/s    0:00:00  (1, 100.0% of 1)                      Fully defragmented!
    
    333333333333333333333333333333333333333333333333333333333333
    ===> Pass 3 of 10 <===    Remaining Fragmentation 0/26 (0%)
    333333333333333333333333333333333333333333333333333333333333
    
    
    444444444444444444444444444444444444444444444444444444444444
    ===> Pass 4 of 10 <===    Remaining Fragmentation 0/26 (0%)
    444444444444444444444444444444444444444444444444444444444444
    
    
    555555555555555555555555555555555555555555555555555555555555
    ===> Pass 5 of 10 <===    Remaining Fragmentation 0/26 (0%)
    555555555555555555555555555555555555555555555555555555555555
    
    
    666666666666666666666666666666666666666666666666666666666666
    ===> Pass 6 of 10 <===    Remaining Fragmentation 0/26 (0%)
    666666666666666666666666666666666666666666666666666666666666
    
    
    777777777777777777777777777777777777777777777777777777777777
    ===> Pass 7 of 10 <===    Remaining Fragmentation 0/26 (0%)
    777777777777777777777777777777777777777777777777777777777777
    
    
    888888888888888888888888888888888888888888888888888888888888
    ===> Pass 8 of 10 <===    Remaining Fragmentation 0/26 (0%)
    888888888888888888888888888888888888888888888888888888888888
    
    
    999999999999999999999999999999999999999999999999999999999999
    ===> Pass 9 of 10 <===    Remaining Fragmentation 0/26 (0%)
    999999999999999999999999999999999999999999999999999999999999
    
    
    101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010
    ===> Pass 10 of 10 <===    Remaining Fragmentation 0/26 (0%)
    101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010101010
    
    ========= COMPLETE ===========
    Remaining Fragmented Files:
    Frags/MB Before:         26.66
    Frags/MB After:          0.00
    Improvement:             100.0 %
    ===============================
    jdong@jdong-laptop:~$

    Attached is a tarball containing both debianized source code (written in Python) and a .deb package for simpler installation. There is only one file (defrag) that needs to be installed. It should be very safe to use (follows the trusted XFS defragger algorithm) on any filesystem, but I don't guarantee that it'll be flawless on every setup. Don't run passes over crucial data unless you have backups, I guess


    The fragmentation numbers printed are in Frags/MB. I think that makes more sense than just counting fragments, so you don't spend time defragging huge files that will naturally be slightly fragmented.


    Enjoy!


    Small Note
    Do _not_ obsessive-compulsively run passes trying to achieve perfection. By default, the defragger counts files with less than .5frags/MB as unfragmented. This can be overridden with the -t parameter. Do it wisely... If you try too hard to defrag, what will happen is that while files may defragment, free space will become fragmented in the process, which will ultimately lower write performance.

    This is not a full defragmenter (i.e. free space consolidation, directory reorganization), but just a file defragger.



    The source code is under the GPL, and you are welcome to play around with it. I'm open to suggestions and contributions. The working directory of the source archive is already a bzr 0.8 knits repository, so it's ready for decentralized commits.

    I will be publishing the head branch as soon as I find a place to host it, as my system will not take the bandwidth hit and Launchpad supermirror does not support Knits yet.
    Attached Files Attached Files
    Last edited by jdong; May 3rd, 2006 at 07:56 PM.
    Quote Originally Posted by tuxradar
    Linux's audio architecture is more like the layers of the Earth's crust than the network model, with lower levels occasionally erupting on to the surface, causing confusion and distress, and upper layers moving to displace the underlying technology that was originally hidden

  2. #2
    Join Date
    May 2006
    Beans
    4

    Re: pyfragtools: Defragmenter for Linux

    O&O has had a defragmenter for Linux for years, i tried it on servers, the database efficiancy went up by... 0%, not measurable.

    You can try this out if you have a recent backup and a lot of time on your hands but the real reason one is not provided in ANY distro is that it's not needed, the improvements aren't measurable.

  3. #3
    Join Date
    Dec 2005
    Beans
    263
    Distro
    Dapper Drake Testing/

    Re: pyfragtools: Defragmenter for Linux

    Hmm. Anything I need to install first?

    Code:
    ImportError: No module named bzrlib.osutils
    Hang in there. It gets better...

  4. #4
    Join Date
    May 2006
    Beans
    4

    Re: pyfragtools: Defragmenter for Linux

    Oh, and XFS is in no way safe to run on PC hardware without a good UPS that will gently shut down your system, that goes for most other filesystems like it.

  5. #5
    Join Date
    Oct 2004
    Location
    Cupertino, CA
    Beans
    5,092
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: pyfragtools: Defragmenter for Linux

    Well, on some mythtv setups, I have seen 100MB files with 7000+ fragments (ext3 & reiser), and at that point read performance degrades to the point that it can't keep up with burning to a 48x CD (approx 5MB/s, on a 40MB/s drive). A simple copy turned it down to 50 or so fragments, and read performance drastically improved.

    That's why there's the built-in limit of 0.5frags/MB, which IMO should be even higher than that.

    As far as system performance, unless you do something special that fragments up the disk (torrenting, mythtv, large downloads), you probably won't realize much, but it's always good to know you're ready for something like this, or if you're curious about fragmentation the analyze mode is great.
    Quote Originally Posted by tuxradar
    Linux's audio architecture is more like the layers of the Earth's crust than the network model, with lower levels occasionally erupting on to the surface, causing confusion and distress, and upper layers moving to displace the underlying technology that was originally hidden

  6. #6
    Join Date
    Oct 2004
    Location
    Cupertino, CA
    Beans
    5,092
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: pyfragtools: Defragmenter for Linux

    Quote Originally Posted by htinn
    Hmm. Anything I need to install first?

    Code:
    ImportError: No module named bzrlib.osutils
    You may need to install bzr. I will try to remove that dependency... The progress bar was stolen from bzrlib
    Quote Originally Posted by tuxradar
    Linux's audio architecture is more like the layers of the Earth's crust than the network model, with lower levels occasionally erupting on to the surface, causing confusion and distress, and upper layers moving to displace the underlying technology that was originally hidden

  7. #7
    Join Date
    May 2006
    Beans
    4

    Re: pyfragtools: Defragmenter for Linux

    Quote Originally Posted by jdong
    Well, on some mythtv setups, I have seen 100MB files with 7000+ fragments (ext3 & reiser), and at that point read performance degrades to the point that it can't keep up with burning to a 48x CD (approx 5MB/s, on a 40MB/s drive). A simple copy turned it down to 50 or so fragments, and read performance drastically improved.

    That's why there's the built-in limit of 0.5frags/MB, which IMO should be even higher than that.

    As far as system performance, unless you do something special that fragments up the disk (torrenting, mythtv, large downloads), you probably won't realize much, but it's always good to know you're ready for something like this, or if you're curious about fragmentation the analyze mode is great.
    You're saying that simultaneous read/write from multiple users on a database server will not fragment like torrenting, mythtv, large downloads?

    We had extreme fragmentation, the real world results of an O&O defragment were not measurable, thus we opted not to implement the function in the nightly runs.

    Provide me with real world benchmarks and i'll trust that it will improve anything, for now it's just another way to potentially screw up my system with no real benefit to be had.

  8. #8
    Join Date
    May 2006
    Beans
    4

    Re: pyfragtools: Defragmenter for Linux

    If you are curious about fragmentation run fsck, if you are worried that it may affect your performance, install more memory, afaik the cache subsystem of Linux is second to none and VERY able to keep up with the demand of burning a cd or dvd, i'd say that burning a cd has more to do with the interface than the ability to deliver when it comes to write speed.

    In 110% of the cases when a burn has to slow down it's because of the writer, the media or the interface, it's never ever because the HDD cannot read (while fully cached) as fast as the burner can write.

    It's not generally known but a fast read doesn't flow to a fast write, both adds to the PCI bus by requests and usually blams the inferface with crap, this happens regardless of fragmentation and is why the QOS of SCSI interfaces is so much smoother, even though it has to be translated.

  9. #9
    Join Date
    Feb 2005
    Location
    Heverlee, Belgium
    Beans
    139
    Distro
    Ubuntu 6.06

    Re: pyfragtools: Defragmenter for Linux

    never mind my original question.
    Nice tool, just what I needed to defrag my disk full of video files.

  10. #10
    Join Date
    Feb 2005
    Location
    Heverlee, Belgium
    Beans
    139
    Distro
    Ubuntu 6.06

    Re: pyfragtools: Defragmenter for Linux

    when I try to run the defrag I get following error message:
    Code:
    fruymen@star1:~$ sudo defrag /media/multimedia/
    Building list of files to analyze... done!
    Traceback (most recent call last):
      File "/usr/sbin/defrag", line 646, in ?
        run(opts[1][0], threshold, passes)
      File "/usr/sbin/defrag", line 549, in run
        pb = ProgressBar(show_pct=True, show_bar=True, show_spinner=True, show_eta=False)
      File "/usr/sbin/defrag", line 31, in ProgressBar
        return TTYProgressBar(to_file=to_file, **kwargs)
      File "/usr/sbin/defrag", line 214, in __init__
        from bzrlib.osutils import terminal_width
    ImportError: cannot import name terminal_width
    any idea how to solve this?
    I already install bzr

Page 1 of 31 12311 ... LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •