Results 1 to 10 of 10

Thread: HOWTO: Install LessFS (Deduplication FS) on 9.10

  1. #1
    Join Date
    Aug 2006
    Location
    Netherlands
    Beans
    37
    Distro
    Ubuntu 10.10 Maverick Meerkat

    Thumbs up HOWTO: Install LessFS (Deduplication FS) on 9.10

    LessFS is a relatively new FS. The primary goal for LessFS is an on-disk backup & archiving filesystem.
    Features of LessFS:
    - On-line Deduplication
    - On-line Compression

    LessFS creates a growable file that represents the LessFS filesystem.
    My configuration is as follows: 5 Harddisks of varying sizes, concatenated to one disk, using LVM2. The data partition formatted as ext4, with large file option set.

    I've installed LessFS as follows:

    Open a Terminal on your server.

    • Get the prerequisites:
      Code:
      sudo apt-get install build-essential libselinux1-dev libsepol1-dev pkg-configlibtokyocabinet-dev checkinstall
    • Get Fuse 2.8.1: download from https://launchpad.net/ubuntu/+source...8.1-1.1ubuntu2 the packages called libfuse, libfuse-dev and fuse-utils. Download the ones that correspond to your architecture.
    • Install them like this:
      Code:
      sudo dpkg -i *.deb
      assuming the FUSE files are in the same folder as the shell is.
    • Download & Extract LessFS: Download .tar.gz package from http://sourceforge.net/projects/lessfs/files/. Extract like
      Code:
      tar xzf *.tar.gz
      Other Option, All-in one (Note: check for the newest version number at wwww.lessfs.com):
      Code:
      wget -O- http://sourceforge.net/projects/lessfs/files/lessfs/lessfs-1.0.1/lessfs-1.0.1.tar.gz/download | tar xz
    • cd into lessfs folder and run
      Code:
      ./configure
      This gives a lot of text. Check for errors.
    • Run
      Code:
      make && sudo checkinstall

    I didn't test checkinstall, so if it fails, replace "checkinstall" with "make install".

    You are done installing. Now configuring:

    LessFS isn't made natively on Debian/Ubuntu. That is to see in the startup scipt. I tweaked it a bit, but I must say it's a quick-and-dirty hack. The tweaked script is provided as an attachment (lessfs.sh).
    Download it and save it to /etc/init.d
    Edit the file if you want to change the mountpoint to something other than /data.
    Then do:
    Code:
    sudo update-rc.d lessfs.sh enable
    This creates the correct startup scripts. This isn't nessecary. You can always hand-start lessfs.

    The next part is important though:
    From the lessfs source folder copy the etc/lessfs.cfg file. Copy it to /etc
    Then edit the new /etc/lessfs.cfg
    change the file paths to your liking (tip: Do the data folder 'dta' as the only one on your large partition. For performance reason)
    Alter the cache settings in a way that it fits your memory.

    After saving the file, create the mta, dta subfolders at the places you stated above.
    Also create the mount point folder.

    do the following to create the folder:
    Code:
    sudo mklessfs -c /etc/lessfs.cfg
    And you're done Start Lessfs with:
    Code:
    sudo invoke-rc.d lessfs.sh start
    or restart.

    Happy Less-ing

    Bubbel

    PS: Thanks to PhracturedBlue for filtering out some errors.
    Attached Files Attached Files
    Last edited by Bubbel; February 1st, 2010 at 10:56 AM. Reason: Small updates

  2. #2
    Join Date
    Apr 2005
    Beans
    11

    Re: HOWTO: Install LessFS (Deduplication FS) on 9.10

    I’d recommend using ’sudo checkinstall’ rather than ’sudo make install’ generally. It makes it easier to uninstall packages in the future.
    I also use x86_64, so those fuse packages didn’t work for me.
    I used the lucid ones instead freom here:
    https://launchpad.net/ubuntu/+source...8.1-1.1ubuntu2
    which worked fine on Karmic (need libfuse, libfuse-dev, fuse-utils)

    I needed this too:
    sudo apt-get install libtokyocabinet-dev

    After that installation went smoothly.
    I didn't find your attachment for auto-starting lessfs, but I don't need that for my use

  3. #3
    Join Date
    Aug 2006
    Location
    Netherlands
    Beans
    37
    Distro
    Ubuntu 10.10 Maverick Meerkat

    Re: HOWTO: Install LessFS (Deduplication FS) on 9.10

    Quote Originally Posted by PhracturedBlue View Post
    I’d recommend using ’sudo checkinstall’ rather than ’sudo make install’ generally. It makes it easier to uninstall packages in the future.
    I also use x86_64, so those fuse packages didn’t work for me.
    I used the lucid ones instead from here:
    https://launchpad.net/ubuntu/+source...8.1-1.1ubuntu2
    which worked fine on Karmic (need libfuse, libfuse-dev, fuse-utils)

    I needed this too:
    sudo apt-get install libtokyocabinet-dev

    After that installation went smoothly.
    I didn't find your attachment for auto-starting lessfs, but I don't need that for my use
    Thank you for the info. I will update the howto. Never heard of checkinstall, so I looked it up. Sounds like a really good solution.
    Greets, Bubbel

  4. #4
    Join Date
    Oct 2006
    Location
    Pittsburgh, PA
    Beans
    69
    Distro
    Ubuntu

    Re: HOWTO: Install LessFS (Deduplication FS) on 9.10

    We want ZFS! :-p

  5. #5
    Join Date
    May 2007
    Location
    Berkeley, CA
    Beans
    3

    Re: HOWTO: Install LessFS (Deduplication FS) on 9.10

    Quote Originally Posted by tlsarles View Post
    We want ZFS! :-p
    Well, unfortunately we aren't going to get it anytime soon, unless you count the FUSE+ZFS hack (It looks amateurish, and I don't think I would trust it).

    This LessFS sounds very interesting though, and I'll check it out soon.

    I've used deduplication on DataDomains & Netapp and it rocks. Two years worth of 300GB backups in a fraction of the space.

    I would love a smaller solution for my home and smaller office systems.

  6. #6
    Join Date
    Mar 2008
    Beans
    16

    Re: HOWTO: Install LessFS (Deduplication FS) on 9.10

    Quote Originally Posted by Bubbel View Post
    LessFS isn't made natively on Debian/Ubuntu. That is to see in the startup scipt. I tweaked it a bit, but I must say it's a quick-and-dirty hack. The tweaked script is provided as an attachment (lessfs.sh).
    Download it and save it to /etc/init.d
    Edit the file if you want to change the mountpoint to something other than /data.
    Then do:
    Code:
    sudo update-rc.d lessfs.sh enable
    This creates the correct startup scripts. This isn't nessecary.
    This did not work for me, however this:
    Code:
    sudo update-rc.d lessfs.sh defaults
    did work.

    After watching the logs after a few reboots, I've decided that my LessFS stores needed to be started last and stopped first, so I went ahead and set mine using:
    Code:
    sudo update-rc.d lessfs.sh defaults 90 10
    On a default ubuntu install this should make lessfs.sh the last script started on boot, and the first script stopped on shutdown. Otherwise I was experiencing messages in my log of "Lessfs has not been unmounted cleanly. Rollback to..."

    Also, your list of apt-get prerequisites needs a space pkg-configlibtokyocabinet-dev. That threw me off for a few minutes until I found the correct names of the two packages.
    Last edited by Tidder; March 28th, 2010 at 08:09 AM.

  7. #7
    Join Date
    Aug 2005
    Location
    Canada, eh?
    Beans
    17

    Re: HOWTO: Install LessFS (Deduplication FS) on 9.10

    Excellent write up - thank you! I've been fighting forever trying to compile TokyoCabinet with no success until finally I find that it's already in the repository!

    Just a note: libfuse2, libfuse-dev and fuse-utils are also available from the Lucid (10.04) repository.

  8. #8
    Join Date
    Jan 2011
    Beans
    2

    Re: HOWTO: Install LessFS (Deduplication FS) on 9.10

    It was an excellent cheat sheet, but I stumbled on the last step. Can anyone help me here?

    I followed the cheatsheet, downloaded lessfs.sh and successfully did mklessfs step as following:

    update-rc.d lessfs.sh defaults
    mklessfs -c /etc/lessfs.cfg

    ( I am running as root, so not adding sudo)

    Then, I ran the following:

    invoke-rc.d lessfs.sh start

    I got the following error:
    fuse: mountpoint is not empty
    fuse: if you are sure this is safe, use the 'nonempty' mount option

    I was trying to use same disk for input and output.
    input: /xfs and output: /xfs/dta and /xfs/mta
    So, I changed and created /xfs2.
    mkdir /xfs2, /xfs2/dta, /xfs2/mta

    # invoke-rc.d lessfs.sh start
    invoke-rc.d: initscript lessfs.sh, action "start" failed.

    So, I ran the individual command as following:

    # lessfs /etc/lessfs.cfg /xfs -o
    kernel_cache,negative_timeout=0,entry_timeout=0,at tr_timeout=0,use_ino,readdir_ino,default_permissio ns,allow_other,big_writes,max_read=131072,max_writ e=131072


    It finished in 1 second without any messages, and I see the following:

    # du -sh /xfs
    4,6M

    #du -sh /xfs2
    73M
    #du -sh /xfs/dta
    9.1M
    #du -sh /xfs/mta
    64M


    I know that I am testing with a small data just to see if I can run the tool successfully before moving into bigger and more realistic data. I would expect /xfs2 to be smaller than /xfs, but it is much much bigger.
    Did the tool run successfully? Why am I seeing results like these?
    And, how do I convert '.tch' files into normal UNIX (xfs) files?

    I really appreciate your help. Thanks a lot in advance,

    Thanks,
    - Seshu

  9. #9
    Join Date
    Jan 2011
    Beans
    2

    Re: HOWTO: Install LessFS (Deduplication FS) on 9.10

    It was an excellent cheat sheet, but I stumbled on the last step. Can anyone help me here?

    I followed the cheatsheet, downloaded lessfs.sh and successfully did mklessfs step as following:

    update-rc.d lessfs.sh defaults
    mklessfs -c /etc/lessfs.cfg

    ( I am running as root, so not adding sudo)

    Then, I ran the following:

    invoke-rc.d lessfs.sh start

    I got the following error:
    fuse: mountpoint is not empty
    fuse: if you are sure this is safe, use the 'nonempty' mount option

    I was trying to use same disk for input and output.
    input: /xfs and output: /xfs/dta and /xfs/mta
    So, I changed and created /xfs2.
    mkdir /xfs2, /xfs2/dta, /xfs2/mta

    # invoke-rc.d lessfs.sh start
    invoke-rc.d: initscript lessfs.sh, action "start" failed.

    So, I ran the individual command as following:

    # lessfs /etc/lessfs.cfg /xfs -o
    kernel_cache,negative_timeout=0,entry_timeout=0,at tr_timeout=0,use_ino,readdir_ino,default_permissio ns,allow_other,big_writes,max_read=131072,max_writ e=131072


    It finished in 1 second without any messages, and I see the following:

    # du -sh /xfs
    4,6M

    #du -sh /xfs2
    73M
    #du -sh /xfs/dta
    9.1M
    #du -sh /xfs/mta
    64M


    I know that I am testing with a small data just to see if I can run the tool successfully before moving into bigger and more realistic data. I would expect /xfs2 to be smaller than /xfs, but it is much much bigger.
    Did the tool run successfully? Why am I seeing results like these?
    And, how do I convert '.tch' files into normal UNIX (xfs) files?

    I really appreciate your help. Thanks a lot in advance,

    Thanks,
    - Seshu

  10. #10
    Join Date
    Sep 2008
    Beans
    1

    Unhappy Re: HOWTO: Install LessFS (Deduplication FS) on 9.10

    Excellent how to.

    Steps to install mhash (I spent some time here for googling)
    ---------------------------------------------------------------

    Get mhash version 0.9.9.9
    wget http://sourceforge.net/projects/mhas....9.9.9.tar.bz2

    Extract
    tar -xjvf mhash-0.9.9.9.tar.bz2

    change the dir to extracted folder of the above step

    ./configure

    make

    Installation

    sudo make install


    Steps to install fuse (again i spent some time with google)
    -----------------------------------------------------------

    Download
    wget https://launchpad.net/ubuntu/+archiv....1.orig.tar.gz

    Extract
    tar -xzvf fuse_2.8.1.orig.tar.gz

    Change the dir to the folder extracted

    ./configure
    make
    sudo make install
    Last edited by hariprasadmce; January 21st, 2011 at 11:53 AM.

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •