Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Rsync needs --ignore-errors to delete files,why is that?

  1. #1
    Join Date
    Jun 2012
    Beans
    182

    Rsync needs --ignore-errors to delete files,why is that?

    I've just found out that all my rsync backups include a large amount (around 35.000 files in total) that are no longer on the source:I have several backups on different drives,and upon restoring one of these on a new hard disk for a test,I spotted some files on the desktop that I had deleted months ago,then I've had a look in my home directory and found lots of stuff that was not supposed to be there anymore-long story short,rsync has not been deleting lots of files that was supposed to delete,in all my backups.

    After some tries using --dry-run to avoid issues,I've discovered that only using the --ignore-errors option I can delete those files,which are mostly old firefox backups and old pictures in my home directory,but there's also stuff in /usr/lib or /var/log that needs to be deleted.

    So,why is this happening and should I run all my backups with the --ignore-errors option from now on? Is it safe?

  2. #2
    Join Date
    Feb 2008
    Beans
    Hidden!

    Re: Rsync needs --ignore-errors to delete files,why is that?

    without knowing what the exact syntax of the rysnc command or script, it's hard to tell.
    Personally, I use and advocate something like
    Code:
    cd ~ && rsync -avz --no-perms . /media/Keepers --exclude "/home/jji/.gvfs" --exclude "/home/jj/.cache/" --exclude "VboxVMs/"  --delete
    Where /media/Keepers is my USB Drive mounted with
    /dev/sdb1 /media/Keepers ntfs-3g rw,uid=1000,gid=100,dmask=022,fmask=133 0 0
    --delete will remove files at the destination that do not exist on the source when it is run.

    This has kept my backups in sync for months, exactly, file for file.
    does your rsync command/script have a --delete option?
    Maybe a combination of --ignore-errors and --delete?

    man rsync for those options or
    http://www.mikerubel.org/computers/rsync_snapshots/

    hope that helps you out.
    Last edited by Habitual; September 20th, 2013 at 04:40 PM.


  3. #3
    Join Date
    Aug 2009
    Location
    Charleston, SC (USA)
    Beans
    1,439
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: Rsync needs --ignore-errors to delete files,why is that?

    It looks like other people have been afflicted with this bug as well.

    Rsync not deleting files on destination

    You might want to use
    Code:
     -n, --dry-run               perform a trial run with no changes made
    Just to make sure nothing is wrong with the new configuration.


    Hope that helps!
    ~Caboose
    UA Team Member
    "When our actions do not, our fears do make us traitors." -- Shakespeare
    "Unless someone like you cares a whole awful lot, nothing is going to get better. It's not!" -- Dr. Seuss
    "No man has ever seen his dreams fulfilled and regretted the sacrifice necessary to fulfill them" -- Mattie Montgomery


  4. #4
    Join Date
    Jun 2012
    Beans
    182

    Re: Rsync needs --ignore-errors to delete files,why is that?

    Thanks for your replies,and sorry for forgetting to post the rsync command I'm using,which is

    Code:
    sudo rsync --delete -avv --progress --exclude='/mnt' --exclude='/media' --exclude='/proc' --exclude='/sys'  --exclude='/dev'  --exclude='/tmp' --exclude='/var/run'  --exclude='lost+found' --exclude='VirtualBox VMs' --exclude='/home/ubuntu/.thumbnails'  / /media/backup_device
    as you can see,the --delete option is obviously there and yet those files which I'm positive are no longer on the source (namely,my current operating system,mostly my home directory) are not being deleted on the destination device:I've done many tests using --dry-run,and apparently the only way to delete such files using the command above is to add to it the --ignore-errors option.

    Unless,I simulate synchronizing only some specific subdirectories inside my home directory (like,say,firefox backup profiles) with their backup copy,in which case file deletion succeeds even without the --ignore-errors option:why is that?

    Is this really some kind of bug in rsync,or is there anything wrong with my command above? I did not see this coming,as I've said I've only discovered it by accident while doing a test of restoring my system on a new hard drive.

    Edit:since I'm doing this from a running system whilst logged in as superuser (using sudo,I mean),would possibly make a difference if I'd do this with all partitions unmounted,i.e. from a live CD or while booted in another OS (I have a multi-boot set up) ?
    Last edited by cogset; September 22nd, 2013 at 09:45 AM.

  5. #5
    Join Date
    Jun 2012
    Beans
    182

    Re: Rsync needs --ignore-errors to delete files,why is that?

    Update:well,of course I did not mean really "unmounted",I rather actually meant "from a non-running system" (accessing partitions / and home from another OS),in which case I can report that interestingly rsync works properly and deletes the old files without needing the -ignore-errors option.

    Come to that,does anybody have a clue about why this happens?

  6. #6
    Join Date
    Jun 2012
    Beans
    182

    Re: Rsync needs --ignore-errors to delete files,why is that?

    I may have found the cause:excluding the directory .gvfs solved the issue,as the IO error that caused rsync to skip file deletion was indeed related to this directory.

    I knew that it was not meant to be backed up,but I wrongly assumed that rsync would be skipping it without side effects-which clearly wasn't the case.

  7. #7
    Join Date
    Feb 2008
    Beans
    Hidden!

    Re: Rsync needs --ignore-errors to delete files,why is that?

    I just checked my target for backups and it has the .gvfs directory, even though I have excluded it.
    Bizarre.

    rsync version 3.0.9 protocol version 30


  8. #8
    Join Date
    Jun 2012
    Beans
    182

    Re: Rsync needs --ignore-errors to delete files,why is that?

    On this system,I have rsync version 3.0.7 protocol version 30 -I might have to check which version I have on Mint 13,that had the exact same issue with rsync.

  9. #9
    Join Date
    Feb 2008
    Beans
    Hidden!

    Re: Rsync needs --ignore-errors to delete files,why is that?

    Well, if it all worked without issue, we'd have nothing to do, heh?

    One thing I see in your command is --exclude='VirtualBox VMs'
    Who's VirtualBox VMs? All? some? someone's?
    You can specify patterns in the /root/rsync.excludes file described below. see
    Code:
    man rsync
    for specifics on
    --exclude-from=FILE read exclude patterns from FILE

    There's an Ubuntu user? Or is that your userid? (sorry, I don't use Ubuntu)

    You may wish to try this approach...stolen from here...
    as root
    Code:
     vi /root/rsync.excludes
    and add

    Code:
    /mnt
    /media
    /proc
    /sys
    /dev
    /tmp
    /var/run
    /lost+found
    /VirtualBox VMs 
    /home/ubuntu/.thumbnails
    and test it with
    Code:
    sudo rsync --delete -avv --dry-run --progress --exclude-from=/root/rsync.excludes /  /media/backup_device
    If it appears to be working as expected, the remove the --dry-run from the command.
    I ran it in dry-run mode and even though it said "Deleting ...." I was pretty nervous seeing it. They were NOT deleted using "dry-run".
    It really is just a dry-run.

    Good luck and I'll check back tomorrow.
    Last edited by Habitual; October 4th, 2013 at 03:50 AM.


  10. #10
    Join Date
    Jun 2012
    Beans
    182

    Re: Rsync needs --ignore-errors to delete files,why is that?

    As for --exclude='VirtualBox VMs' ,that is a pattern to exclude my whole VirtualBox VMs directory,as I couldn't figure out how to quote (for the embedded space) and enclose in --exclude=' ' at the same time...it works,so I've left it like that.

    Talking about the --exclude-from=FILE approach,that's interesting and definitely neater than writing a lot of exclusion rules (indeed,I may give it a shot and make a dry run test),but what other advantage may it give in this specific case? Given that apparently the culprit was the .gvfs directory,and excluding it makes rsync work as it should,I take there would be no difference in using a file or manually writing the exclusion rules one by one ?

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •