You and rsync - what's your backup use

[BBQdave]

Long story short: A long long time ago I had a salvaged 500GB HD that I put in a USB Case. I copy Documents folder and other data to that HD - drag and drop. Over the years of doing that I have several (time labeled) folders of data. An example would be DOCs230228 which is Documents folder from 2023-02-28.

After years of this, I have multiple time labeled folders of data, most the same data recopied. I am still not even close to using 1/4 of that HD.

So would rsync benefit me? Copy and sync only changed data? Or keep on keeping on with time labeled folders and data saved for that time period?

Your thoughts and experiences appreciated

[TheFu]

This is why a backup tool that handles versioning would be better for you. rdiff-backup works using librync, and the command really isn't THAT different.

With just a little more planning, you can get a much better backup that is useful for rebuilding the system in less than 30-45 minutes to be just like it was before. https://ubuntuforums.org/showthread....2#post13928432 has a simple script. The only things I'd change in that script now, for a really simple backup, are

/usr/bin/rdiff-backup  --exclude-special-files  ....

becomes

/usr/bin/rdiff-backup  --exclude-sockets --exclude-device-files --exclude-fifos ...

"special files" also includes symbolic links, so excluding those isn't good for everyone.

And I'd add

/usr/bin/apt-mark showauto > /root/backups/apt-mark.showauto

with the other apt-mark command. Turns out that both list of packages can be needed due to the way that APT dependencies happen.

[BBQdave]

Appreciate the information TheFu. More to read, more to explore

[TheFu]

Appreciate the information TheFu. More to read, more to explore

When testing new backup methods and tools, can I suggest using /etc/ as the directory? It has files that need to have specific permissions, specific owners and specific groups, while being relatively tiny. Modifying a file in /etc/ can be done with little risk to see how versioning backup tools handle it. Adding a line with a comment in /etc/hosts is an easy thing, right? Adding a new file is also easy.

So, with just a few backups ... perhaps 3-5, and minor changes between each backup set, we can see how those changes are handled to gain comfort with each tool.

At restore time, we can easily fine 50MB of space under /tmp/ to restore into and see that owners, groups, permissions, ACLs, xattrs are all maintained. Heck, add a few symbolic links (/etc/resolv.conf is usually a symlink) and you have a test for everything important. If your backup tool handles /etc/ well, it will handle your HOME well too, probably.

One of the main reasons I like rdiff-backup is because the most recent backup seems just like an rsync mirror. When I need to restore a file, 99% of the time, I need the most recent backup from last night, so the restore is just a copy or scp or sftp. Easy for anyone to understand. If I need older versions, only then do I need to look up the rdiff-backup -r {timespec} command timespec. It is very flexible - last week, 13 days ago, 2 months ago, 2 backup-sets ago, or an exact date in yyyy-mm-dd format, if that's needed.

Backing up /etc/ is just one of those things that is extremely useful even if we can't directly restore everything there. Having those files as reference on a new system **is** useful and if you put comments with specific tags in the config files that you've modified over the days, weeks, years under /etc/, then you'll be able to relocated those files and changes again quickly.

$ cd /etc/
$ sudo fgrep -ril   thefu  * |more
aliases
aliases.db
chrony/chrony.conf
cron.daily/00logwatch
fstab
hosts
logwatch/conf/services/zz-disk_space.conf.53475.2023-03-06@10:28:27~
NetworkManager/system-connections/Wired connection 1.nmconnection
postfix/main.cf
resolv.conf
samba/win7lap-D.credentials
ssh/sshd_config
ssl/certs/ca-certificates.crt

So I don't have to hunt down too many files in /etc/ thanks to tagging I've done on that system. I do use ansible, so many files in /etc/ don't have specific tags added, but are custom. Rerunning the ansible devops tool will fix those. Sadly, the systemctl commands used to mask 20 things don't get captured. Sigh.
That's a relatively new system, less than a month old, so only the core stuff is there.

[maglin2]

I've used luckybackup for a few years.
It's just a frontend for rsync, but it makes like easier for numpties like me.
Most of my data is static, so each run takes a few seconds.
I used to run backintime (which I think was based on rdiff-backup), and was very happy with it till it stopped working, and it was beyond me to figure out why. Don't know what it's like now.
Edit - it seems backintime was based on rsync too.

[TheFu]

back-in-time uses hardlinks for the different backup sets. This is very different from how rdiff-backup works, which uses reverse differential files that are compressed to minimize the storage needed for file differences.
The hardlink method of backup versioning, as used by rsnapshot, back-in-time, and 500 other script front-ends, has been around for 30+ yrs and it works well, with a few liabilities. Almost every Unix Admin book has a script that uses hardlinks in an appendix.

Almost all the free backup tools on Unix-like OSes are based on librsync. rsync is a great tool. I use it multiple times every day, but not for backups where I need versioning. There are better, easier, tools, for that.

Everyone talks about backups and that's good, but backups without the ability to restore are useless. That's where looking at the backup stroage areas and understanding how the files are actually stored makes a huge difference. I like to see the files, see they are there, obviously correct with the right owner, group, permissions, ACLs and xattrs, because when restoring system files, all those things are critical. A simple copy doesn't handle those details correctly.
There are backup tools that throw data into specific chunk sizes unrelated to the actual files included in the backup. They capture all the other stuff beyond just the file and they can even encrypt those chunks so each can be stored safely on someone else's computer disks. That's how duplicity, Deja Dup, Duplicati work. But without one of those tools, there's no getting the data back. Whereas with back-in-time or rdiff-backup, the files are stored in what turns out to be a mirror of the source directory tree. Restoring a file is a copy or ... an rsync. Which every is easier. For me, that's a key difference. The fact that rdiff-backup commands look very much like rsync commands is another bonus.

[zebra2]

For me LuckyBackup works just fine. I will have to admit that during the Wayland transition when LuckyBackup had compatibility problems rsync served me pretty well but I never did get it doing exactly what I wanted. With rsync I was spending too much time on management without the best results. Years back when I was a hotshot brainiac I probably would have been ok with rsync. Not today.

[BBQdave]

Thanks again all for the information and experience.
I am researching and gaining an education

[aljames2]

+1 for rdiff-backup. Love it.

I don't like image type backup solutions for anything, seldom do they restore correctly, especially if your hardware has changed, and with most of them you cannot go retrieve a file without restoring or upacking the entire image. As stated, rdiff-backup allows you to restore from previous versions, or you can just go get the most recent version of a single file you may need.

@TheFu... very cool idea to place a unique # comment in all custom files, scripts, etc. and using that nifty search idea. Simple and awesome! Thx for that!

[maglin2]

By the way, bit of an eggs in one basket paranoia thing I suppose, but I also run the default deja dup backup to a separate drive (as well as luckybackup).
It's a useful fallback, but its product is not readily inspectable just from a file manager. The luckybackup product is. I usually know what file I most recently added/modded and always check for that in backup after running.