rdiff-backup: how to exclude hidden files and folders ?

**freeflyjohn** · August 3rd, 2021

I am using rdiff-backup to backup my Mac home folder to my Ubuntu server.

I am trying to exclude hidden files and folders (i.e. starting with a '.') using the --exclude-filelist option and specifying a text file.

For testing purposes I have created these dummy folders (i.e. empty) and files on the Mac in /Users/myname/rdiff_test/source...

The paths specified in the rdiff-backup command are:

EXCLUDE LIST: /Users/myname/.rdiff-backup/exclude_list.txt

SOURCE: /Users/myname/rdiff_test/source

DESTINATION: /Users/myname/rdiff_test/backup

The rdiff-backup command I run is:

Code:

rdiff-backup --exclude-filelist  /Users/myname/.rdiff-backup/exclude_list.txt /Users/myname/rdiff_test/source /Users/myname/rdiff_test/backup

The file exclude_list.txt contains:

Code:

/Users/myname/rdiff_test/source/*.Trash
/Users/myname/rdiff_test/source/*.AppleDouble
/Users/myname/rdiff_test/source/*$RECYCLE.BIN
/Users/myname/rdiff_test/source/*.ini
/Users/myname/rdiff_test/source/.*
/Users/myname/rdiff_test/source/iperf3
/Users/myname/rdiff_test/source/Applications
/Users/myname/rdiff_test/source/Applications (Parallels)
+ /Users/myname/rdiff_test/source/Desktop
/Users/myname/rdiff_test/source/Documents/Microsoft User Data
+ /Users/myname/rdiff_test/source/Documents
/Users/myname/rdiff_test/source/Downloads/svn
+ /Users/myname/rdiff_test/source/Downloads
/Users/myname/rdiff_test/source/Library
+ /Users/myname/rdiff_test/source/Movies
+ /Users/myname/rdiff_test/source/Music
/Users/myname/rdiff_test/source/Nextcloud
+ /Users/myname/rdiff_test/source/Pictures 
/Users/myname/rdiff_test/source/rdiff_test
/Users/myname/rdiff_test/source/Trash
/Users/myname/rdiff_test/source/Parallels

I was hoping the following line would have excluded the hidden files and folders but it hasn't:

Code:

/Users/myname/rdiff_test/source/.*

What's the syntax required in the exclude file list to exclude these hidden folders and files ?

**GhX6GZMB** · August 3rd, 2021

From the man page, I'd say you need to use --exclude-globbing-filelist instead of just --exclude-filelist in your command.

**TheFu** · August 4th, 2021

So, I tried to answer with my guess and the forum software is blocking it. Sorry.

I've never used --exclude-filelist. I work from the opposite method. Exclude everything, then specifically include the stuff I want.
Also, I pull backups, never push them for security reasons.

Code:

 rdiff-backup --exclude-special-files  \
               client-hostname::/home/username /backup/client
 rdiff-backup --remove-older-than 90d --force \
               /backup/client

is the basic "pull" method.

For backing up systems that can be restored to a fresh install, things are a little more complex, some setup is required to put the system data into places included in the backup and to setup a backup userid that is root-equivalent, but the rdiff-backup commands look like this:

Code:

sudo rdiff-backup
       --exclude-special-files \
       --exclude '**/.cache/mozilla' \
       --exclude '**/.cache/chromium' \
       --include /usr/local --include /etc --include /home --include /root \
       --exclude '**'   backup3288@${remote-machine}::/ \ 
       /Backups/${remote-machine}

sudo  rdiff-backup --remove-older-than 90d --force        /Backups/${remote-machine}

The order of the include/excludes matters in the precedence.

I know ZERO about OSX besides generalities. Had an OSX machine for a few weeks and before it was smashed against the wall over frustration, I gave it to a coworker.

**freeflyjohn** · August 4th, 2021

Originally Posted by TheFu

Also, I pull backups, never push them for security reasons.

Code:

 rdiff-backup --exclude-special-files  \
               client-hostname::/home/username /backup/client
 rdiff-backup --remove-older-than 90d --force \
               /backup/client

is the basic "pull" method.

Thanks TheFu

Just to clarify, I am only 'testing' rdiff-backup on the Mac, using local source and destination folders, simply to get the syntax correct for excluding files.

Once I get the syntax correct, I will be pulling a backup of the Mac from the server (via SSH) by running this command from the server:

Code:

rdiff-backup -v5 --exclude-filelist ~/.rdiff-backup/exclude_list.txt myname@macbookpro.fritz.box::/Users/myname /media/storage/Backup/Mac/rdiff-backup/myname

rdiff-backup took about 14 hours to backup 680Gb from the Mac when I first ran this command from the server, but the exclude options did not work correctly.

This is why I am testing locally on the Mac first and using empty folders as it takes seconds to test whether the exclude options work or not.

I am not interested in backing up system files, just personal data.

**freeflyjohn** · August 4th, 2021

Originally Posted by ml9104

From the man page, I'd say you need to use --exclude-globbing-filelist instead of just --exclude-filelist in your command.

Thanks ml9104

I did initially use the --exclude-globbing-filelist option when I performed the first backup using the following:

Code:

**.Spotlight
**.Trash
**.AppleDouble
**$RECYCLE.BIN
**.ini
**/.*
**iperf3
**/Applications
**/Applications\ (Parallels)
+ **/Desktop
**/Documents/Microsoft User Data
**/Documents/Parallels
+ **/Documents
**/Downloads/svn
+ **/Downloads
**/Library
+ **/Movies
+ **/Music
**/Nextcloud
**/Parallels
+ **/Pictures
**/Public
**/Trash

It correctly excluded the hidden files and folders, but it excluded any folder called 'Applications'.

I only wanted to exclude the one folder called 'Applications' at the top level in the users home folder to be excluded, not every single folder called 'Applications' !

For example, here is the source which has a folder called 'Applications' in the 'Pictures' folder:

And here is the backup, where the 'Applications' folder in the 'Pictures' folder has not been copied (when I want it to be included):

This is the only 'Applications' folder I want to be excluded, not any other folders by the same name:

This is why I tried the --exclude-filelist option in the hope I could explicitly exclude flles and folders using full paths (like I used to do with rsycn).

I don't quite understand this globbing syntax, its all new to me.

**TheFu** · August 4th, 2021

**/Applications

matches any parent directories with 'Applications' in the name. The '**' says to match any number of parent directories - 1, 2, 3, 200, 900 .... doesn't matter.

**freeflyjohn** · August 4th, 2021

Originally Posted by TheFu

**/Applications

matches any parent directories with 'Applications' in the name. The '**' says to match any number of parent directories - 1, 2, 3, 200, 900 .... doesn't matter.

So how do I resolve this ?

If I use the --exclude-filelist option I can define the following so it only excludes the 'Applications' folder at the top level which I want to exclude...

Code:

/Users/myname/rdiff_test/source/Applications

But then I cant seem to use the --exclude-filelist option to exclude the hidden files and folders at the top level.

So how do I use the --exclude-globbing-filelist option to only exclude only the 'Applications' folder at the top level, not every single folder called 'Applications' ?

**freeflyjohn** · August 4th, 2021

Think I've solved it.

I can define absolute paths for the folders I want to include/exclude as shown below and this seems to work !

Code:

**.Spotlight
**.Trash
**.AppleDouble
**$RECYCLE.BIN
**.ini
/Users/myname/rdiff_test/source/.*
/Users/myname/rdiff_test/source/iperf3
/Users/myname/rdiff_test/source/Applications
/Users/myname/rdiff_test/source/Applications (Parallels)
+ /Users/myname/rdiff_test/source/Desktop
/Users/myname/rdiff_test/source/Documents/Microsoft User Data
/Users/myname/rdiff_test/source/Documents/Parallels
+ /Users/myname/rdiff_test/source/Documents
/Users/myname/rdiff_test/source/Downloads/svn
+ /Users/myname/rdiff_test/source/Downloads
/Users/myname/rdiff_test/source/Library
+ /Users/myname/rdiff_test/source/Movies
+ /Users/myname/rdiff_test/source/Music
/Users/myname/rdiff_test/source/Nextcloud
/Users/myname/rdiff_test/source/Parallels
+ /Users/myname/rdiff_test/source/Pictures
/Users/myname/rdiff_test/source/Public
/Users/myname/rdiff_test/source/Trash

**freeflyjohn** · August 4th, 2021

I'm so confused by this exclude list because the moment I change the order of any of the lines, depending on the order it either includes everything or excludes everything.

At the moment it seems to work correctly if the order is as shown below:

Code:

**.Spotlight
**.Trash
**.AppleDouble
**$RECYCLE.BIN
**.ini
/Users/myname/rdiff_test/source/.*
/Users/myname/rdiff_test/source/iperf3
/Users/myname/rdiff_test/source/Applications
/Users/myname/rdiff_test/source/Applications (Parallels)
+ /Users/myname/rdiff_test/source/Desktop
/Users/myname/rdiff_test/source/Documents/Microsoft User Data
/Users/myname/rdiff_test/source/Documents/Parallels
+ /Users/myname/rdiff_test/source/Documents
/Users/myname/rdiff_test/source/Downloads/svn
+ /Users/myname/rdiff_test/source/Downloads
/Users/myname/rdiff_test/source/Library
+ /Users/myname/rdiff_test/source/Movies
+ /Users/myname/rdiff_test/source/Music
/Users/myname/rdiff_test/source/Nextcloud
/Users/myname/rdiff_test/source/Parallels
+ /Users/myname/rdiff_test/source/Pictures
/Users/myname/rdiff_test/source/Public
/Users/myname/rdiff_test/source/Trash

But if the order is changed as shown below, it includes everything:

Code:

**.AppleDouble
**.ini
**.Spotlight
**.Trash
**$RECYCLE.BIN
/Users/myname/rdiff_test/source/.*
/Users/myname/rdiff_test/source/Applications
/Users/myname/rdiff_test/source/Applications (Parallels)
/Users/myname/rdiff_test/source/Documents/Microsoft User Data
/Users/myname/rdiff_test/source/Documents/Parallels
/Users/myname/rdiff_test/source/Downloads/svn
/Users/myname/rdiff_test/source/iperf3
/Users/myname/rdiff_test/source/Library
/Users/myname/rdiff_test/source/Nextcloud
/Users/myname/rdiff_test/source/Parallels
/Users/myname/rdiff_test/source/Public
/Users/myname/rdiff_test/source/Trash
+ /Users/myname/rdiff_test/source/Desktop
+ /Users/myname/rdiff_test/source/Documents
+ /Users/myname/rdiff_test/source/Downloads
+ /Users/myname/rdiff_test/source/Movies
+ /Users/myname/rdiff_test/source/Music
+ /Users/myname/rdiff_test/source/Pictures

The same applies if its changed to the order below:

Code:

+ /Users/myname/rdiff_test/source/Desktop
+ /Users/myname/rdiff_test/source/Documents
+ /Users/myname/rdiff_test/source/Downloads
+ /Users/myname/rdiff_test/source/Movies
+ /Users/myname/rdiff_test/source/Music
+ /Users/myname/rdiff_test/source/Pictures
**.AppleDouble
**.ini
**.Spotlight
**.Trash
**$RECYCLE.BIN
/Users/myname/rdiff_test/source/.*
/Users/myname/rdiff_test/source/Applications
/Users/myname/rdiff_test/source/Applications (Parallels)
/Users/myname/rdiff_test/source/Documents/Microsoft User Data
/Users/myname/rdiff_test/source/Documents/Parallels
/Users/myname/rdiff_test/source/Downloads/svn
/Users/myname/rdiff_test/source/iperf3
/Users/myname/rdiff_test/source/Library
/Users/myname/rdiff_test/source/Nextcloud
/Users/myname/rdiff_test/source/Parallels
/Users/myname/rdiff_test/source/Public
/Users/myname/rdiff_test/source/Trash

It makes absolutely no sense how the change in order completely breaks the exclude function ???

I know TheFu said 'the order of the include/excludes matters in the precedence' which is why I tried moving the + lines at the top and at the bottom, but it made no difference.

I'm getting extremely frustrated with rdiff-backup and seriously beginning to dislike it, purely down to these exclude issues. It doesn't help that the manual is poor (https://rdiff-backup.net/docs/rdiff-backup.1.html)

For example, in the manual it states there is an option called "--exclude-special-files", but there is no explanation anywhere in the manual as to what this option does.

I might go back to using rsync at this rate because with rdiff-backup I have no confidence its including and excluding the correct files and folders.

At least with rsync its straight forward to include and exclude files and folders, unlike rdiff-backup

**TheFu** · August 4th, 2021

From the manpage:

Code:

       --exclude-special-files
              Exclude all device files, fifo files, socket  files,  and  sym‐
              bolic links.

If that isn't clear, then you are lacking some basic knowledge. It is extremely clear to administrators.

What manual are you using if not the manpage on your system, for the exact version of rdiff-backup being used? The local manpage should be the primary source of information for every command.

I've already said to exclude everything and include what you want backed up. For some reason you've decided to reject that idea. Good luck.

Thread: rdiff-backup: how to exclude hidden files and folders ?

Thread Tools

Display

rdiff-backup: how to exclude hidden files and folders ?

Re: rdiff-backup: how to exclude hidden files and folders ?

Re: rdiff-backup: how to exclude hidden files and folders ?

Re: rdiff-backup: how to exclude hidden files and folders ?

Re: rdiff-backup: how to exclude hidden files and folders ?

Re: rdiff-backup: how to exclude hidden files and folders ?

Re: rdiff-backup: how to exclude hidden files and folders ?

Re: rdiff-backup: how to exclude hidden files and folders ?

Re: rdiff-backup: how to exclude hidden files and folders ?

Re: rdiff-backup: how to exclude hidden files and folders ?

Bookmarks

Bookmarks

Posting Permissions