Page 1 of 2 12 LastLast
Results 1 to 10 of 15

Thread: You and rsync - what's your backup use

  1. #1
    Join Date
    Jun 2011
    Location
    North Carolina
    Beans
    514
    Distro
    Ubuntu

    Question You and rsync - what's your backup use

    Long story short: A long long time ago I had a salvaged 500GB HD that I put in a USB Case. I copy Documents folder and other data to that HD - drag and drop. Over the years of doing that I have several (time labeled) folders of data. An example would be DOCs230228 which is Documents folder from 2023-02-28.

    After years of this, I have multiple time labeled folders of data, most the same data recopied. I am still not even close to using 1/4 of that HD.

    So would rsync benefit me? Copy and sync only changed data? Or keep on keeping on with time labeled folders and data saved for that time period?

    Your thoughts and experiences appreciated
    Ubuntu 24.04 LTS | Toshiba Satellite C655 | i3 2.3Ghz | Intel HD Graphics 3000 | 8GB RAM | 65GB SSD
    Fedora 40 | Lenovo Edge 15 | i5 1.7Ghz | Intel HD Graphics 4400 | 6GB RAM | 1TB HDD

  2. #2
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: You and rsync - what's your backup use

    This is why a backup tool that handles versioning would be better for you. rdiff-backup works using librync, and the command really isn't THAT different.

    With just a little more planning, you can get a much better backup that is useful for rebuilding the system in less than 30-45 minutes to be just like it was before. https://ubuntuforums.org/showthread....2#post13928432 has a simple script. The only things I'd change in that script now, for a really simple backup, are
    Code:
    /usr/bin/rdiff-backup  --exclude-special-files  ....
    becomes
    Code:
    /usr/bin/rdiff-backup  --exclude-sockets --exclude-device-files --exclude-fifos ...
    "special files" also includes symbolic links, so excluding those isn't good for everyone.

    And I'd add
    Code:
    /usr/bin/apt-mark showauto > /root/backups/apt-mark.showauto
    with the other apt-mark command. Turns out that both list of packages can be needed due to the way that APT dependencies happen.

  3. #3
    Join Date
    Jun 2011
    Location
    North Carolina
    Beans
    514
    Distro
    Ubuntu

    Re: You and rsync - what's your backup use

    Appreciate the information TheFu. More to read, more to explore
    Ubuntu 24.04 LTS | Toshiba Satellite C655 | i3 2.3Ghz | Intel HD Graphics 3000 | 8GB RAM | 65GB SSD
    Fedora 40 | Lenovo Edge 15 | i5 1.7Ghz | Intel HD Graphics 4400 | 6GB RAM | 1TB HDD

  4. #4
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: You and rsync - what's your backup use

    Quote Originally Posted by BBQdave View Post
    Appreciate the information TheFu. More to read, more to explore
    When testing new backup methods and tools, can I suggest using /etc/ as the directory? It has files that need to have specific permissions, specific owners and specific groups, while being relatively tiny. Modifying a file in /etc/ can be done with little risk to see how versioning backup tools handle it. Adding a line with a comment in /etc/hosts is an easy thing, right? Adding a new file is also easy.

    So, with just a few backups ... perhaps 3-5, and minor changes between each backup set, we can see how those changes are handled to gain comfort with each tool.

    At restore time, we can easily fine 50MB of space under /tmp/ to restore into and see that owners, groups, permissions, ACLs, xattrs are all maintained. Heck, add a few symbolic links (/etc/resolv.conf is usually a symlink) and you have a test for everything important. If your backup tool handles /etc/ well, it will handle your HOME well too, probably.

    One of the main reasons I like rdiff-backup is because the most recent backup seems just like an rsync mirror. When I need to restore a file, 99% of the time, I need the most recent backup from last night, so the restore is just a copy or scp or sftp. Easy for anyone to understand. If I need older versions, only then do I need to look up the rdiff-backup -r {timespec} command timespec. It is very flexible - last week, 13 days ago, 2 months ago, 2 backup-sets ago, or an exact date in yyyy-mm-dd format, if that's needed.

    Backing up /etc/ is just one of those things that is extremely useful even if we can't directly restore everything there. Having those files as reference on a new system **is** useful and if you put comments with specific tags in the config files that you've modified over the days, weeks, years under /etc/, then you'll be able to relocated those files and changes again quickly.
    Code:
    $ cd /etc/
    $ sudo fgrep -ril   thefu  * |more
    aliases
    aliases.db
    chrony/chrony.conf
    cron.daily/00logwatch
    fstab
    hosts
    logwatch/conf/services/zz-disk_space.conf.53475.2023-03-06@10:28:27~
    NetworkManager/system-connections/Wired connection 1.nmconnection
    postfix/main.cf
    resolv.conf
    samba/win7lap-D.credentials
    ssh/sshd_config
    ssl/certs/ca-certificates.crt
    So I don't have to hunt down too many files in /etc/ thanks to tagging I've done on that system. I do use ansible, so many files in /etc/ don't have specific tags added, but are custom. Rerunning the ansible devops tool will fix those. Sadly, the systemctl commands used to mask 20 things don't get captured. Sigh.
    That's a relatively new system, less than a month old, so only the core stuff is there.

  5. #5
    Join Date
    Feb 2014
    Beans
    298

    Re: You and rsync - what's your backup use

    I've used luckybackup for a few years.
    It's just a frontend for rsync, but it makes like easier for numpties like me.
    Most of my data is static, so each run takes a few seconds.
    I used to run backintime (which I think was based on rdiff-backup), and was very happy with it till it stopped working, and it was beyond me to figure out why. Don't know what it's like now.
    Edit - it seems backintime was based on rsync too.
    Last edited by maglin2; March 8th, 2023 at 10:49 AM.

  6. #6
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: You and rsync - what's your backup use

    back-in-time uses hardlinks for the different backup sets. This is very different from how rdiff-backup works, which uses reverse differential files that are compressed to minimize the storage needed for file differences.
    The hardlink method of backup versioning, as used by rsnapshot, back-in-time, and 500 other script front-ends, has been around for 30+ yrs and it works well, with a few liabilities. Almost every Unix Admin book has a script that uses hardlinks in an appendix.

    Almost all the free backup tools on Unix-like OSes are based on librsync. rsync is a great tool. I use it multiple times every day, but not for backups where I need versioning. There are better, easier, tools, for that.

    Everyone talks about backups and that's good, but backups without the ability to restore are useless. That's where looking at the backup stroage areas and understanding how the files are actually stored makes a huge difference. I like to see the files, see they are there, obviously correct with the right owner, group, permissions, ACLs and xattrs, because when restoring system files, all those things are critical. A simple copy doesn't handle those details correctly.
    There are backup tools that throw data into specific chunk sizes unrelated to the actual files included in the backup. They capture all the other stuff beyond just the file and they can even encrypt those chunks so each can be stored safely on someone else's computer disks. That's how duplicity, Deja Dup, Duplicati work. But without one of those tools, there's no getting the data back. Whereas with back-in-time or rdiff-backup, the files are stored in what turns out to be a mirror of the source directory tree. Restoring a file is a copy or ... an rsync. Which every is easier. For me, that's a key difference. The fact that rdiff-backup commands look very much like rsync commands is another bonus.

  7. #7
    Join Date
    May 2020
    Location
    top of the mountain
    Beans
    190
    Distro
    Ubuntu Development Release

    Re: You and rsync - what's your backup use

    For me LuckyBackup works just fine. I will have to admit that during the Wayland transition when LuckyBackup had compatibility problems rsync served me pretty well but I never did get it doing exactly what I wanted. With rsync I was spending too much time on management without the best results. Years back when I was a hotshot brainiac I probably would have been ok with rsync. Not today.

  8. #8
    Join Date
    Jun 2011
    Location
    North Carolina
    Beans
    514
    Distro
    Ubuntu

    Re: You and rsync - what's your backup use

    Thanks again all for the information and experience.
    I am researching and gaining an education
    Ubuntu 24.04 LTS | Toshiba Satellite C655 | i3 2.3Ghz | Intel HD Graphics 3000 | 8GB RAM | 65GB SSD
    Fedora 40 | Lenovo Edge 15 | i5 1.7Ghz | Intel HD Graphics 4400 | 6GB RAM | 1TB HDD

  9. #9
    Join Date
    Feb 2019
    Location
    Virginia
    Beans
    386
    Distro
    Xubuntu 22.04 Jammy Jellyfish

    Re: You and rsync - what's your backup use

    +1 for rdiff-backup. Love it.

    I don't like image type backup solutions for anything, seldom do they restore correctly, especially if your hardware has changed, and with most of them you cannot go retrieve a file without restoring or upacking the entire image. As stated, rdiff-backup allows you to restore from previous versions, or you can just go get the most recent version of a single file you may need.

    @TheFu... very cool idea to place a unique # comment in all custom files, scripts, etc. and using that nifty search idea. Simple and awesome! Thx for that!

  10. #10
    Join Date
    Feb 2014
    Beans
    298

    Re: You and rsync - what's your backup use

    By the way, bit of an eggs in one basket paranoia thing I suppose, but I also run the default deja dup backup to a separate drive (as well as luckybackup).
    It's a useful fallback, but its product is not readily inspectable just from a file manager. The luckybackup product is. I usually know what file I most recently added/modded and always check for that in backup after running.

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •