Page 1 of 3 123 LastLast
Results 1 to 10 of 23

Thread: rdiff-backup: how to exclude hidden files and folders ?

  1. #1
    Join Date
    Nov 2017
    Beans
    101

    rdiff-backup: how to exclude hidden files and folders ?

    I am using rdiff-backup to backup my Mac home folder to my Ubuntu server.

    I am trying to exclude hidden files and folders (i.e. starting with a '.') using the --exclude-filelist option and specifying a text file.

    For testing purposes I have created these dummy folders (i.e. empty) and files on the Mac in /Users/myname/rdiff_test/source...




    The paths specified in the rdiff-backup command are:

    EXCLUDE LIST: /Users/myname/.rdiff-backup/exclude_list.txt
    SOURCE: /Users/myname/rdiff_test/source
    DESTINATION: /Users/myname/rdiff_test/backup

    The rdiff-backup command I run is:

    Code:
    rdiff-backup --exclude-filelist  /Users/myname/.rdiff-backup/exclude_list.txt /Users/myname/rdiff_test/source /Users/myname/rdiff_test/backup
    The file exclude_list.txt contains:

    Code:
    /Users/myname/rdiff_test/source/*.Trash
    /Users/myname/rdiff_test/source/*.AppleDouble
    /Users/myname/rdiff_test/source/*$RECYCLE.BIN
    /Users/myname/rdiff_test/source/*.ini
    /Users/myname/rdiff_test/source/.*
    /Users/myname/rdiff_test/source/iperf3
    /Users/myname/rdiff_test/source/Applications
    /Users/myname/rdiff_test/source/Applications (Parallels)
    + /Users/myname/rdiff_test/source/Desktop
    /Users/myname/rdiff_test/source/Documents/Microsoft User Data
    + /Users/myname/rdiff_test/source/Documents
    /Users/myname/rdiff_test/source/Downloads/svn
    + /Users/myname/rdiff_test/source/Downloads
    /Users/myname/rdiff_test/source/Library
    + /Users/myname/rdiff_test/source/Movies
    + /Users/myname/rdiff_test/source/Music
    /Users/myname/rdiff_test/source/Nextcloud
    + /Users/myname/rdiff_test/source/Pictures 
    /Users/myname/rdiff_test/source/rdiff_test
    /Users/myname/rdiff_test/source/Trash
    /Users/myname/rdiff_test/source/Parallels
    I was hoping the following line would have excluded the hidden files and folders but it hasn't:

    Code:
    /Users/myname/rdiff_test/source/.*
    What's the syntax required in the exclude file list to exclude these hidden folders and files ?
    Attached Images Attached Images

  2. #2
    Join Date
    Jun 2019
    Location
    Dirndl-land
    Beans
    851
    Distro
    Lubuntu 20.04 Focal Fossa

    Re: rdiff-backup: how to exclude hidden files and folders ?

    From the man page, I'd say you need to use --exclude-globbing-filelist instead of just --exclude-filelist in your command.

  3. #3
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: rdiff-backup: how to exclude hidden files and folders ?

    So, I tried to answer with my guess and the forum software is blocking it. Sorry.

    I've never used --exclude-filelist. I work from the opposite method. Exclude everything, then specifically include the stuff I want.
    Also, I pull backups, never push them for security reasons.
    Code:
     rdiff-backup --exclude-special-files  \
                   client-hostname::/home/username /backup/client
     rdiff-backup --remove-older-than 90d --force \
                   /backup/client
    is the basic "pull" method.

    For backing up systems that can be restored to a fresh install, things are a little more complex, some setup is required to put the system data into places included in the backup and to setup a backup userid that is root-equivalent, but the rdiff-backup commands look like this:
    Code:
    sudo rdiff-backup
           --exclude-special-files \
           --exclude '**/.cache/mozilla' \
           --exclude '**/.cache/chromium' \
           --include /usr/local --include /etc --include /home --include /root \
           --exclude '**'   backup3288@${remote-machine}::/ \ 
           /Backups/${remote-machine}
    
    sudo  rdiff-backup --remove-older-than 90d --force        /Backups/${remote-machine}
    The order of the include/excludes matters in the precedence.

    I know ZERO about OSX besides generalities. Had an OSX machine for a few weeks and before it was smashed against the wall over frustration, I gave it to a coworker.

  4. #4
    Join Date
    Nov 2017
    Beans
    101

    Re: rdiff-backup: how to exclude hidden files and folders ?

    Quote Originally Posted by TheFu View Post
    Also, I pull backups, never push them for security reasons.
    Code:
     rdiff-backup --exclude-special-files  \
                   client-hostname::/home/username /backup/client
     rdiff-backup --remove-older-than 90d --force \
                   /backup/client
    is the basic "pull" method.
    Thanks TheFu

    Just to clarify, I am only 'testing' rdiff-backup on the Mac, using local source and destination folders, simply to get the syntax correct for excluding files.

    Once I get the syntax correct, I will be pulling a backup of the Mac from the server (via SSH) by running this command from the server:

    Code:
    rdiff-backup -v5 --exclude-filelist ~/.rdiff-backup/exclude_list.txt myname@macbookpro.fritz.box::/Users/myname /media/storage/Backup/Mac/rdiff-backup/myname
    rdiff-backup took about 14 hours to backup 680Gb from the Mac when I first ran this command from the server, but the exclude options did not work correctly.

    This is why I am testing locally on the Mac first and using empty folders as it takes seconds to test whether the exclude options work or not.

    I am not interested in backing up system files, just personal data.

  5. #5
    Join Date
    Nov 2017
    Beans
    101

    Re: rdiff-backup: how to exclude hidden files and folders ?

    Quote Originally Posted by ml9104 View Post
    From the man page, I'd say you need to use --exclude-globbing-filelist instead of just --exclude-filelist in your command.
    Thanks ml9104

    I did initially use the --exclude-globbing-filelist option when I performed the first backup using the following:

    Code:
    **.Spotlight
    **.Trash
    **.AppleDouble
    **$RECYCLE.BIN
    **.ini
    **/.*
    **iperf3
    **/Applications
    **/Applications\ (Parallels)
    + **/Desktop
    **/Documents/Microsoft User Data
    **/Documents/Parallels
    + **/Documents
    **/Downloads/svn
    + **/Downloads
    **/Library
    + **/Movies
    + **/Music
    **/Nextcloud
    **/Parallels
    + **/Pictures
    **/Public
    **/Trash
    It correctly excluded the hidden files and folders, but it excluded any folder called 'Applications'.

    I only wanted to exclude the one folder called 'Applications' at the top level in the users home folder to be excluded, not every single folder called 'Applications' !

    For example, here is the source which has a folder called 'Applications' in the 'Pictures' folder:




    And here is the backup, where the 'Applications' folder in the 'Pictures' folder has not been copied (when I want it to be included):




    This is the only 'Applications' folder I want to be excluded, not any other folders by the same name:




    This is why I tried the --exclude-filelist option in the hope I could explicitly exclude flles and folders using full paths (like I used to do with rsycn).

    I don't quite understand this globbing syntax, its all new to me.
    Attached Images Attached Images
    Last edited by freeflyjohn; August 4th, 2021 at 10:57 AM.

  6. #6
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: rdiff-backup: how to exclude hidden files and folders ?

    **/Applications

    matches any parent directories with 'Applications' in the name. The '**' says to match any number of parent directories - 1, 2, 3, 200, 900 .... doesn't matter.

  7. #7
    Join Date
    Nov 2017
    Beans
    101

    Re: rdiff-backup: how to exclude hidden files and folders ?

    Quote Originally Posted by TheFu View Post
    **/Applications

    matches any parent directories with 'Applications' in the name. The '**' says to match any number of parent directories - 1, 2, 3, 200, 900 .... doesn't matter.
    So how do I resolve this ?

    If I use the --exclude-filelist option I can define the following so it only excludes the 'Applications' folder at the top level which I want to exclude...

    Code:
    /Users/myname/rdiff_test/source/Applications
    But then I cant seem to use the --exclude-filelist option to exclude the hidden files and folders at the top level.

    So how do I use the --exclude-globbing-filelist option to only exclude only the 'Applications' folder at the top level, not every single folder called 'Applications' ?

  8. #8
    Join Date
    Nov 2017
    Beans
    101

    Re: rdiff-backup: how to exclude hidden files and folders ?

    Think I've solved it.

    I can define absolute paths for the folders I want to include/exclude as shown below and this seems to work !

    Code:
    **.Spotlight
    **.Trash
    **.AppleDouble
    **$RECYCLE.BIN
    **.ini
    /Users/myname/rdiff_test/source/.*
    /Users/myname/rdiff_test/source/iperf3
    /Users/myname/rdiff_test/source/Applications
    /Users/myname/rdiff_test/source/Applications (Parallels)
    + /Users/myname/rdiff_test/source/Desktop
    /Users/myname/rdiff_test/source/Documents/Microsoft User Data
    /Users/myname/rdiff_test/source/Documents/Parallels
    + /Users/myname/rdiff_test/source/Documents
    /Users/myname/rdiff_test/source/Downloads/svn
    + /Users/myname/rdiff_test/source/Downloads
    /Users/myname/rdiff_test/source/Library
    + /Users/myname/rdiff_test/source/Movies
    + /Users/myname/rdiff_test/source/Music
    /Users/myname/rdiff_test/source/Nextcloud
    /Users/myname/rdiff_test/source/Parallels
    + /Users/myname/rdiff_test/source/Pictures
    /Users/myname/rdiff_test/source/Public
    /Users/myname/rdiff_test/source/Trash

  9. #9
    Join Date
    Nov 2017
    Beans
    101

    Re: rdiff-backup: how to exclude hidden files and folders ?

    I'm so confused by this exclude list because the moment I change the order of any of the lines, depending on the order it either includes everything or excludes everything.

    At the moment it seems to work correctly if the order is as shown below:

    Code:
    **.Spotlight
    **.Trash
    **.AppleDouble
    **$RECYCLE.BIN
    **.ini
    /Users/myname/rdiff_test/source/.*
    /Users/myname/rdiff_test/source/iperf3
    /Users/myname/rdiff_test/source/Applications
    /Users/myname/rdiff_test/source/Applications (Parallels)
    + /Users/myname/rdiff_test/source/Desktop
    /Users/myname/rdiff_test/source/Documents/Microsoft User Data
    /Users/myname/rdiff_test/source/Documents/Parallels
    + /Users/myname/rdiff_test/source/Documents
    /Users/myname/rdiff_test/source/Downloads/svn
    + /Users/myname/rdiff_test/source/Downloads
    /Users/myname/rdiff_test/source/Library
    + /Users/myname/rdiff_test/source/Movies
    + /Users/myname/rdiff_test/source/Music
    /Users/myname/rdiff_test/source/Nextcloud
    /Users/myname/rdiff_test/source/Parallels
    + /Users/myname/rdiff_test/source/Pictures
    /Users/myname/rdiff_test/source/Public
    /Users/myname/rdiff_test/source/Trash
    But if the order is changed as shown below, it includes everything:

    Code:
    **.AppleDouble
    **.ini
    **.Spotlight
    **.Trash
    **$RECYCLE.BIN
    /Users/myname/rdiff_test/source/.*
    /Users/myname/rdiff_test/source/Applications
    /Users/myname/rdiff_test/source/Applications (Parallels)
    /Users/myname/rdiff_test/source/Documents/Microsoft User Data
    /Users/myname/rdiff_test/source/Documents/Parallels
    /Users/myname/rdiff_test/source/Downloads/svn
    /Users/myname/rdiff_test/source/iperf3
    /Users/myname/rdiff_test/source/Library
    /Users/myname/rdiff_test/source/Nextcloud
    /Users/myname/rdiff_test/source/Parallels
    /Users/myname/rdiff_test/source/Public
    /Users/myname/rdiff_test/source/Trash
    + /Users/myname/rdiff_test/source/Desktop
    + /Users/myname/rdiff_test/source/Documents
    + /Users/myname/rdiff_test/source/Downloads
    + /Users/myname/rdiff_test/source/Movies
    + /Users/myname/rdiff_test/source/Music
    + /Users/myname/rdiff_test/source/Pictures
    The same applies if its changed to the order below:

    Code:
    + /Users/myname/rdiff_test/source/Desktop
    + /Users/myname/rdiff_test/source/Documents
    + /Users/myname/rdiff_test/source/Downloads
    + /Users/myname/rdiff_test/source/Movies
    + /Users/myname/rdiff_test/source/Music
    + /Users/myname/rdiff_test/source/Pictures
    **.AppleDouble
    **.ini
    **.Spotlight
    **.Trash
    **$RECYCLE.BIN
    /Users/myname/rdiff_test/source/.*
    /Users/myname/rdiff_test/source/Applications
    /Users/myname/rdiff_test/source/Applications (Parallels)
    /Users/myname/rdiff_test/source/Documents/Microsoft User Data
    /Users/myname/rdiff_test/source/Documents/Parallels
    /Users/myname/rdiff_test/source/Downloads/svn
    /Users/myname/rdiff_test/source/iperf3
    /Users/myname/rdiff_test/source/Library
    /Users/myname/rdiff_test/source/Nextcloud
    /Users/myname/rdiff_test/source/Parallels
    /Users/myname/rdiff_test/source/Public
    /Users/myname/rdiff_test/source/Trash
    It makes absolutely no sense how the change in order completely breaks the exclude function ???

    I know TheFu said 'the order of the include/excludes matters in the precedence' which is why I tried moving the + lines at the top and at the bottom, but it made no difference.

    I'm getting extremely frustrated with rdiff-backup and seriously beginning to dislike it, purely down to these exclude issues. It doesn't help that the manual is poor (https://rdiff-backup.net/docs/rdiff-backup.1.html)

    For example, in the manual it states there is an option called "--exclude-special-files", but there is no explanation anywhere in the manual as to what this option does.

    I might go back to using rsync at this rate because with rdiff-backup I have no confidence its including and excluding the correct files and folders.

    At least with rsync its straight forward to include and exclude files and folders, unlike rdiff-backup

  10. #10
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: rdiff-backup: how to exclude hidden files and folders ?

    From the manpage:

    Code:
           --exclude-special-files
                  Exclude all device files, fifo files, socket  files,  and  sym‐
                  bolic links.
    If that isn't clear, then you are lacking some basic knowledge. It is extremely clear to administrators.

    What manual are you using if not the manpage on your system, for the exact version of rdiff-backup being used? The local manpage should be the primary source of information for every command.

    I've already said to exclude everything and include what you want backed up. For some reason you've decided to reject that idea. Good luck.

Page 1 of 3 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •