Page 1 of 2 12 LastLast
Results 1 to 10 of 18

Thread: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

  1. #1
    Join Date
    Jun 2011
    Beans
    14

    5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    I am running Ubuntu 10.04.
    I am transferring roughly 62 GB of data libraries to my 84 GB /home partition.

    I'm using rsync because scp kept stalling, and I had to restart it over and over.

    Things were going great until recently when it began to show an error: "failed: No space left on device (28)"
    These are the things I've done so far:


    • Used the GUI to find out how much I have copied so far: 5,149,552 which take up 30.2 GB.


    • df -h, it tells me that my /home partition is 56% used, and that I have 33 GB available.

    (42 GB used out of 78 GB with 33 GB available)
    Also, none of my other partitions are anywhere near 100%. the /home partition is the most-used and it's only a little over half-full.
    • du -s in the directory where I'm copying all of this: it also returned 42206500.


    • Additionally, when I try to save screen captures, it sometimes fails with a "device full" error.


    What's going on? Am I really out of space? Why doesn't it show me that I'm out of space?

    Is there a hidden temp file that rsync uses that just got too full?

    I did a little research on wikipedia and it said that ext4 has a 64,000 directory limit. Could it be that I somehow broke that limit with all of these files?

    Any help would be appreciated.
    Thank you all for your time,
    Noobinabox


    EDIT:

    Solution: not enough inodes for the vast amount of subdirectories on hard drive. This wasn't an RSYNC problem, rather, a partition configuration issue.
    To check inode usage: df -i
    If you want to add any inodes, you will need to backup your partition and format it using mke2fs (man mke2fs). Be sure to change the respective inode setting.
    Last edited by noobinabox; June 17th, 2011 at 05:17 AM. Reason: Put solution on first post.

  2. #2
    Join Date
    Mar 2010
    Location
    ρθΦ=000
    Beans
    Hidden!
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    Quote Originally Posted by noobinabox View Post
    df -h, it tells me that my /home partition is 56% used, and that I have 33 GB available.
    (42 GB used out of 78 GB with 33 GB available)
    You have 33GB available but are trying to transfer 62GB?

  3. #3
    Join Date
    Jun 2011
    Beans
    14

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    Quote Originally Posted by seawolf167 View Post
    You have 33GB available but are trying to transfer 62GB?
    No, I have 33 GB available after what I've transferred so far (30.2 GB)

    So, I did a du -s on the directory where I'm pulling the files from, and it returned about 62 GB.

    I began the transfer, and got 30.2 GB transferred according to the GUI. Now, rsync is telling me that there is no room on the device even though df -h reported that I had 33 GB still available in /home.

    I apologize for any lack of clarity. Hope that clears it up.

  4. #4
    Join Date
    Mar 2010
    Location
    ρθΦ=000
    Beans
    Hidden!
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    Iff the files you are transferring are all contained in their own directory tree on the source and iff you want to copy them to their own separate directory tree on the destination, what is the result of the following command (properly modified)?

    Code:
    rsync -r -u -v --delete-before /source/directory/tree /destination/directory/tree

  5. #5
    Join Date
    Jun 2011
    Beans
    14

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    Code:
    rsync -r -u -v --delete-before /source/directory/tree /destination/directory/tree
    What exactly does --delete-before delete? I'd like to preserve my source files, so I want to make sure I'm not deleting anything on the source.

    Also I think I should mention that I have been using this command

    Code:
    rsync -vaur [source directory] [destination directory]

  6. #6
    Join Date
    Mar 2010
    Location
    ρθΦ=000
    Beans
    Hidden!
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    The --delete-before deletes extraneous files/folders from the destination that are not present in the source. In the previous command, since the -r switch is used, this applies to all subdirectories as well. Essentially, all this command does is mirror one directory tree to another, subsequent runs updates the files/directories as needed and keep the destination tree exactly as your source tree is

    For more info

    Code:
    man rsync

  7. #7
    Join Date
    Jun 2011
    Beans
    14

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    I've been running rsync for a while now.

    It is currently "receiving file list..."

    In the past, when I used rsync -ruav --progress, it said "receiving incremental file list..." and that took up to 20 min.

    I will let you know what it outputs when it receives the file list.

  8. #8
    Join Date
    Mar 2010
    Location
    ρθΦ=000
    Beans
    Hidden!
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    If you add the -n switch it will do a dryrun and tell you what will be transferred without actual actually transferring anything. You must have a lot of files for it to take 20 min to transfer the file list.

  9. #9
    Join Date
    Jun 2011
    Beans
    14

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    It's basically around 50-60 GB consisting of mostly 100 KB files.

    Hm, the command rsync -r -u -v --delete-before doesn't seem to be able to get out of "receiving file list."

    I cancelled my last attempt and tried rsync -ruav --progress and it transferred a few files before giving me the error
    Code:
    failed: No space left on device (28)
    I restarted rsync -r -u -v --delete-before again and will run it overnight to see what happens. Thank you for all of your help so far, seawolf167. I appreciate you investing your time in helping me.

    This is the code I'm using:
    Code:
    rsync -r -u -v --delete-before username@server:/home/share/sourcedirectory .
    I'm in the directory which I want to copy the contents of sourcedirectory when i run this.


    EDIT:
    So it built overnight:

    output is as follows:

    Multiple lines of code with following error:
    Code:
    rsync: mkstemp "/home/destinationdirectory" failed: No space left on device (28)
    And, at the end:
    Code:
    sent 142437 bytes  received 151494021 bytes  8356.70 bytes/sec
    total size is 29433959618  speedup is 194.11
    rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1526) [generator=3.0.7]
    df -h still reports the following:
    Code:
    Filesystem            Size  Used Avail Use% Mounted on
    /dev/sda6              14G  4.2G  8.9G  32% /
    none                  2.0G  348K  2.0G   1% /dev
    none                  2.0G  240K  2.0G   1% /dev/shm
    none                  2.0G   84K  2.0G   1% /var/run
    none                  2.0G     0  2.0G   0% /var/lock
    none                  2.0G     0  2.0G   0% /lib/init/rw
    /dev/sda7              78G   42G   33G  56% /home
    Last edited by noobinabox; June 10th, 2011 at 01:15 PM.

  10. #10
    Join Date
    Mar 2010
    Location
    ρθΦ=000
    Beans
    Hidden!
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: 5,149,552 files in 30.2 GB and RSYNC says I'm out of space when I still have some

    Quote Originally Posted by noobinabox View Post
    It's basically around 50-60 GB consisting of mostly 100 KB files.[/CODE]
    This is the part I'm concerned is stopping rsync. I have used rsync successfully to transfer 5TB of data from server-to-server (of 1MB -> 1GB files), but I'm wondering if your large file list is causing problems. I would see if you can transfer the files in multiple segments so handling a file list of that size (~500k files?) isn't an issue. Maybe work from the bottom up or transfer your directory tree as a number of smaller subtrees?

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •