Results 1 to 7 of 7

Thread: Rsync: why different number of files in destination vs. source?

  1. #1
    Join Date
    Apr 2014
    Beans
    84

    Rsync: why different number of files in destination vs. source?

    Hi, I used rsync to backup my home directory to an external USB hard drive. Both drives are ext4 filesystem.

    The command I used was:
    rsync -a --progress /home/brent /media/brent/Elements/Lubuntu20^04_backups/Home_rsync_bkp

    When I examined both the source and destination folders to compare them (using PCManFM's "properties" fuction) there was a slight difference in the number of files and and space occupied by the source and copy:

    SOURCE:
    Contains: 69,533 files

    DESTINATION:
    Contains: 69,525 files

    I've attached screenshots below. Click to enlarge them.
    Just wondering why this is the case? Thanks.

    destination.jpg source.jpg

    NOTE: While rsync was copying the files, I did open and edit a document in a text editor (FeatherPad), but I didn't save the file. I'm wondering if that could have caused some log files or something to have been created by Featherpad that would have been written to the source and explain the discrepancy.
    Last edited by Brent_Santin; March 28th, 2021 at 04:53 PM.

  2. #2
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Rsync: why different number of files in destination vs. source?

    rsync doesn't delete files in the target, unless that is specifically requested.

    GUI programs, and almost all editors, commonly create temporary files in the same directory as the file being edited.

    To backup files for a real backup, it is almost always best to use a backup tool that includes versioning.

    rsnapshot is in the ubuntu repos and has been well-tested for 20+ yrs. It is based on rsync.
    The target storage needs to support normal Unix features, so don't use NTFS or anything with "FAT" in the name for the file system.

    If you want to ensure something like what you are doing cannot happen, that's where LVM or ZFS with snapshots comes in. A quiesced file system ensures no corruption for files which may have been "open()" during the copy time. LVM snapshots + something like rdiff-backup are really excellent for efficient, fast, versioned, backups, but you can use LVM snapshots + rsync to get 1 clean copy too, if you prefer. Versioned backups aren't a 1-for-1 storage thing. Here's what rdiff-backup does with 90 days of daily, versioned, backups:
    https://ubuntuforums.org/showthread....2#post13932882
    Basically, 9G of files in the backup used 10.1G for 90days of backups. About 10% more storage for all those versions? Seems like a no-brainer.

  3. #3
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    2,942
    Distro
    Ubuntu Development Release

    Re: Rsync: why different number of files in destination vs. source?

    Try having another look with rsync, to see if it can provide any insight.

    For example, and for this reply, I did a backup of my local directory to another disk, albeit on the same computer. For number of files, I get:

    Code:
    doug@s19:/media/sdb/backup/s19$ find . -type f | wc -l
    313253
    doug@s19:/media/sdb/backup/s19$ find /home/doug -type f | wc -l
    313255
    Oh, the number of files differs by 2. I wonder why? Well, maybe rsync can tell me (note the use of --dry-run, as I don't actually want to do it again):

    Code:
    doug@s19:/media/sdb/backup/s19$ rsync --delete --archive --verbose --dry-run /home/doug ./
    sending incremental file list
    doug/.bash_history
    doug/vm/desk-ff.img
    doug/vm/serv-xx.img
    
    sent 9,847,891 bytes  received 71,198 bytes  3,967,635.60 bytes/sec
    total size is 203,185,919,579  speedup is 20,484.33 (DRY RUN)
    Well, of course my bash_history changed, as I had typed new commands. And oh ya, those two virtual machine image files are root owned:


    Code:
    doug@s19:/media/sdb/backup/s19$ ls -l /home/doug/vm
    total 25690068
    -rw------- 1 root root 53695545344 Nov 21 13:06 desk-ff.img
    -rw------- 1 root root 53695545344 Mar  5 14:23 serv-xx.img
    EDIT: I guess I was off doing this reply when TheFu replied. And yes, I had it backwards that your destination had more files than the source. Notice that I used the --delete option.
    Last edited by Doug S; March 28th, 2021 at 07:10 PM.
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

  4. #4
    Join Date
    Apr 2014
    Beans
    84

    Re: Rsync: why different number of files in destination vs. source?

    Okay, sounds like the text editor use probably caused the change in file numbers.

    rsync doesn't delete files in the target, unless that is specifically requested.
    This was from a source to a totally blank drive, so I don't think the --delete parameter would have helped in this case.

  5. #5
    Join Date
    Feb 2011
    Location
    Coquitlam, B.C. Canada
    Beans
    2,942
    Distro
    Ubuntu Development Release

    Re: Rsync: why different number of files in destination vs. source?

    Quote Originally Posted by Brent_Santin View Post
    Okay, sounds like the text editor use probably caused the change in file numbers.
    So, verify that assumption. Example, where I deleted 4 files and did it again:

    Code:
    doug@s19:/media/sdb/backup/s19$ rsync --delete --archive --verbose --dry-run /home/doug ./
    sending incremental file list
    deleting doug/turbostatfx2
    deleting doug/turbostatfx1
    deleting doug/turbostatdq
    deleting doug/turbostatd
    doug/
    doug/.bash_history
    doug/vm/desk-ff.img
    doug/vm/serv-xx.img
    
    sent 9,847,782 bytes  received 71,286 bytes  3,967,627.20 bytes/sec
    total size is 203,185,362,788  speedup is 20,484.32 (DRY RUN)
    Any follow-up information on your issue would be appreciated. Please have the courtesy to report back.

  6. #6
    Join Date
    Oct 2005
    Location
    Lab, Slovakia
    Beans
    10,337

    Re: Rsync: why different number of files in destination vs. source?

    You could use ls to make a list of each directory, then compare the two lists with diff. Then you will know exactly what is different.

  7. #7
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Rsync: why different number of files in destination vs. source?

    Quote Originally Posted by HermanAB View Post
    You could use ls to make a list of each directory, then compare the two lists with diff. Then you will know exactly what is different.
    Or use diff/sdiff to compare the directory "files" - or meld or dircmp. This takes advantage of the "everything is a file in Unix" fundamental view. It turns out that directories are just files too.
    https://unix.stackexchange.com/quest...ies-using-diff

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •