Page 2 of 2 FirstFirst 12
Results 11 to 18 of 18

Thread: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

  1. #11
    Join Date
    Jan 2010
    Beans
    14

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    Try the --inplace option and let me know what you get. I have a folder structure with > 1M small files and I use rsync as a backup tool. The command line I use to sync up the folder structure is

    rsync -av --delete --progress --inplace /home/user/ /tb/homedir-backup/
    Once you get it going like you want, ditch the --progress switch. The --delete switch deletes from the target, in my case that's the /tb/homedir-backup/ portion.

  2. #12
    Join Date
    Jun 2011
    Beans
    14

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    I did a few more tests, and this is what happened:

    I tried creating a test.txt using "touch test.txt"
    Code:
    touch: cannot touch `test.txt': No space left on device
    Then I tried scp on a test.txt that I created on the server.
    Code:
    ./test.txt: No space left on device
    How can this be? With my very limited knowledge of Linux, I can only guess one of the following has happened:

    1. That ext4 memory has file/subdirectory limits, and that I have reached those limits.


    • If so, it must be based off of partition size since the server can store it.
    1. Or, I am indeed out of space, and df -h is reporting my free disk space incorrectly.

    Regardless, I believe rsync is in fact working correctly, and that this must be some issue with my system's configuration.
    Last edited by noobinabox; June 10th, 2011 at 05:03 PM.

  3. #13
    Join Date
    Jan 2010
    Beans
    14

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    Time to check the filesystem.

    sudo umount /dev/sdb1
    sudo fsck.ext4 -Dfv -E fragcheck /dev/sdb1
    Umount the volume. Be sure to substitute /dev/sdb1 with whatever is appropriate for your system.

    For the fsck command:
    -D optimizes the directories on the filesystem.
    -f forces the check even if the filesystem seems clean.
    -v tells you what's going on.
    -E fragcheck During pass 1, print a detailed report of any discontiguous blocks for files in the filesystem.

    Go get a sandwich and wait for it to complete. I do this every time I add/delete a bunch of files/directories.

  4. #14
    Join Date
    Jun 2011
    Beans
    14

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    I think I found the source of my problems:

    Apparently, ext4 filesystem has unlimited scalability (up to 4 billion) for the number of inodes it can have. However, there is a default inode count for every file system when it is created.

    so I ran a df -i and got this:
    Code:
    Filesystem            Inodes   IUsed   IFree IUse% Mounted on
    /dev/sda6             915712  195691  720021   22% /
    none                  506099     866  505233    1% /dev
    none                  507229       7  507222    1% /dev/shm
    none                  507229      47  507182    1% /var/run
    none                  507229       1  507228    1% /var/lock
    none                  507229       1  507228    1% /lib/init/rw
    /dev/sda7            5152768 5152768       0  100% /home
    I did a little more reading and discovered that inode count cannot be altered after the filesystem has been created.

    So this means I will have to format /home and recreate it with the new inode count, right?

    One question on backing up files:

    I have a large external backup that uses NTFS filesystem. Can I back up my ext4 files to that drive and still get them back afterwards (i.e. In the transfer, will the files transform from ext4 to NTFS? and vice versa)?

    My plan is to use rsync -av to copy the files over.

  5. #15
    Join Date
    Jan 2010
    Beans
    14

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    I transfer files back and forth between ext4 and ntfs all the time. You may need to tweak the permissions once you push the files back over to ext4 depending on whether you use Samba, NFS, USB drive, etc as your method to get the files back over. I presume all these files are owned by the same user.
    Last edited by patmagee1024; June 14th, 2011 at 02:36 PM.

  6. #16
    Join Date
    Apr 2005
    Location
    Finland/UK
    Beans
    Hidden!
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    You definitely can move files between Ext and NTFS, but as NTFS doesn't support Linux-style fle ownerships and permisisons you might run into some troubles after you copy the files back.

    One way to work around that is to put your files into a .tar archive and copy that to your NTFS partition.
    Last edited by mcduck; June 14th, 2011 at 02:52 PM.

  7. #17
    Join Date
    Jun 2011
    Beans
    14

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    Alright, everything worked out well, I think.

    I didn't think about putting it all in a .tar, so naturally file permissions were all wack when I copied them back over to ext4.

    Used these commands I found from google.

    Code:
    $ sudo chown user:user /home/user/.ICEauthority
    $ sudo chmod 644 /home/user/.ICEauthority
    $ sudo chown -R user /home/user
    to fix permissions and ownership.
    And now everything seems to be working fine.

    Inode count is 15 mil now, so that should work out fine. I'd consider this solved. Thanks, everyone, for the great tips and such.

  8. #18
    Join Date
    Feb 2011
    Location
    Milwaukee, WI
    Beans
    7
    Distro
    Ubuntu 11.04 Natty Narwhal

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    couldn't agree more with the guys saying to use -n when tweaking / testing what you're trying to achieve with rsync.
    the --exclude is very handy too if you want to move a bunch of nested data and use some pattern matching to limit what you want to leave out.
    it's just the tip of the iceberg but i did a quick little write up on it with an example here -> http://www.chadstovern.com/copy-all-...ern-with-rsync

Page 2 of 2 FirstFirst 12

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •