Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: Questions on my backup strategy

  1. #1
    Join Date
    Feb 2014
    Beans
    14

    Questions on my backup strategy

    So I've just built a new machine, running 21.04 currently, that will double as a plex server and home file server. It will serve out samba shares so my family will save all their documents to that network location rather than on their personal pc's or laptops. My question comes to my backup strategy. My strategy is to backup the files from this server to a couple of external hard drives I have right now. In addition to that I will be backing up to AWS Glacier for long term disaster recovery, in the case that my house floods or burns down and I lose both our original and backup drives. (Hoping never to have to retrieve from that, but you never know) Anyway, in searching for a decent method for local backups, I am leaning towards a simple rsync script, like the one I found on this page.

    I like it because it's simple and will keep dated incremental backups using the minimal amount of space possible. The question comes with how far back to keep. I am backing up a lot of old family photos and videos. Most of which will never change, so the dates are very old on them. I'm not a backup admin, and obviously am not super familiar with how this should work. But I'm going to end up with a lot of dated folders for backups. The thing I don't want to happen is for the backups of all those old photos to get deleted just because they are past the retention period for backups I am going for. So say I want my retention period to be 60 days. Say I have backup folders going back 60 days like this:

    2021-06-17_22:00
    2021-06-18_22:00
    2021-06-19_22:00
    2021-06-19_22:00
    ...
    ... and so on

    If I set a cron job to delete folders older than 60 days, and it goes to delete that folder from Jun 17th, am I going to lose any backups of all my photos? Say that June 17th was my first full backup, the rest were incrementals. If I delete that first folder, then I've deleted my full backup, and the incrementals are kind of worthless right?

    So do I need to take a full backup once in a while, and only delete backups older than the oldest full backup? I don't have a ton of space on my external drives at the moment. I'd like to build a NAS as a dedicated backup server some time, but I just spent my cash on this new file server, so I'm having to make due with the space I have. So I'm trying to utilize as little space for my backups as possible. But just trying to wrap my head around this so I don't end up with a flawed backup philosophy. Sorry for the noob questions, just hoping I can come up with a simple solid backup strategy. Thanks!

  2. #2
    Join Date
    May 2010
    Beans
    1,701

    Re: Questions on my backup strategy

    If they are diffs then you will need the previous day's backup to know the difference. You could ask the creator of the script or make a new backup job with some dummy files and have a play there to see what happens. You can use the "touch" command to make new files each day, then when you have a few days, delete one of the older backups then attempt a restore, see if it works

  3. #3
    Join Date
    Feb 2014
    Beans
    14

    Re: Questions on my backup strategy

    Quote Originally Posted by ActionParsnip View Post
    If they are diffs then you will need the previous day's backup to know the difference. You could ask the creator of the script or make a new backup job with some dummy files and have a play there to see what happens. You can use the "touch" command to make new files each day, then when you have a few days, delete one of the older backups then attempt a restore, see if it works
    I can modify the script no problem, and yeah I have been creating dummy files to test with, just haven't really tested with those old files. I guess I can just do some testing.

  4. #4
    Join Date
    Oct 2005
    Location
    Lab, Slovakia
    Beans
    10,442

    Re: Questions on my backup strategy

    If you are interested, here is how I do differential backups:
    5 moves, 1 copy and 1 rsync, over SSH for safety.

  5. #5
    aeronutt is offline Grande Half-n-Half Cinnamon Ubuntu
    Join Date
    Feb 2009
    Location
    Central MD
    Beans
    838

    Re: Questions on my backup strategy

    Partial / incremental backups are exactly what you need to minimize storage space. And especially for remote backups, minimize network load.
    I switched from an rsync script to borg backup a few years ago. It adds easy control over full or incremental, also adds data compression, encryption, signatures, if needed.
    I won't backup on a remote location without encryption.
    Compression can take longer to backup than uncompressed if you're local. But for remote with slow network speeds, compression can help a lot.
    FYI, dejadup used to have a bug that caused large files to take forever to backup, which was the main reason I chose borg.

    The main reason to do a 'new' full backup vs a partial, is the risk of data corruption on the storage media. Mean Time Between corruption, total data size, etc has to be taken in to account.

    I'd probably just recommend something like a full backup once a year, and partials at a frequency depending on how often your data changes, and how important loss of 1 month of data would be, for example.

    With borg, you can delete outdated partial or full backups, so it's easy to manage limited disk space.

  6. #6
    Join Date
    Oct 2005
    Location
    Lab, Slovakia
    Beans
    10,442

    Re: Questions on my backup strategy

    Note that many backup systems are overly complicated, in order to compensate for using a bad file system on your storage server. For example, deduplication and compression are built-in features of both Btrfs and Zfs (dedupe can also be done on Ext4 with a periodic deduplication batch file). Encryption on Linux systems should be layered on with LUKS. Transport security over the network could be done with SSH. So, if your storage server uses Btrfs with deduplication and it uses whole disk encryption with LUKS and you run rsync over SSH, then your backup script can be very simple. In this case, you can copy everything as a series of full backups and delete very old backups as required - the file system will handle the deduplication and compression. TIMTOWTDI and TINSTAAFL apply...
    Last edited by HermanAB; August 21st, 2021 at 08:30 PM.

  7. #7
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Questions on my backup strategy

    rsnapshot is in the Ubuntu repos and does rsync + hardlinks for versioned backups.
    Back-in-Time is in the Ubuntu repos and does rsync + hardlinks for versioned backups - with a GUI.
    rdiff-backup is in the Ubuntu repos and does rsync + reverse incremental diff.gz files. This is what I use.

    No need to reinvent the wheel.

    Depending on the types of files and how often they change, different backup methods above will have different results.

    BTW, it appears you should review what a hardlink is before doing anything else. There are some caveats in using them if multiple disks are involved in the backups. rdiff-backup doesn't have those same limitations, but it has some other caveats.

    These tools are all well tested and work. rsnapshot is closest to what that script does.

    I keep between 90 and 180 days of daily backups with rdiff-backup.

    My suggestion is to test each backup tool for a few days and include backing up /home and /etc, so you know what is and isn't possible. Look at the target backup area and see what gets created. That will open your eyes greatly too.

  8. #8
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Questions on my backup strategy

    rsnapshot is in the Ubuntu repos and does rsync + hardlinks for versioned backups.
    Back-in-Time is in the Ubuntu repos and does rsync + hardlinks for versioned backups - with a GUI.
    rdiff-backup is in the Ubuntu repos and does rsync + reverse incremental diff.gz files. This is what I use.

    No need to reinvent the wheel.

    Depending on the types of files and how often they change, different backup methods above will have different results.

    BTW, it appears you should review what a hardlink is before doing anything else. There are some caveats in using them if multiple disks are involved in the backups. rdiff-backup doesn't have those same limitations, but it has some other caveats.

    These tools are all well tested and work. rsnapshot is closest to what that script does.

    I keep between 90 and 180 days of daily backups with rdiff-backup.

    My suggestion is to test each backup tool for a few days and include backing up /home and /etc, so you know what is and isn't possible. Look at the target backup area and see what gets created. That will open your eyes greatly too.

    Some links for ref:

  9. #9
    Join Date
    Apr 2014
    Beans
    800

    Re: Questions on my backup strategy

    I've been tinkering with a Duplicity script that I have backing up various things on my system. Easy for any user to restore their files from via the Deja-Dup gui. I'm liking it.

    I've tested restorations from it. Thus far it's working well. Right now it's backing up to a local drive. Easily modified for ssh over the network and any folders you want. Quite efficient on space it seems as well.

    https://gitlab.com/jmgibson1981/home...ocalbackups.sh
    Last edited by Tadaen_Sylvermane; August 21st, 2021 at 11:40 PM.

  10. #10
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Questions on my backup strategy

    Duplicity checks all the most desired features for excellent backups. In theory, duplicity, Duplicati and DejaDup are all compatible.

    There are 2 downsides with these.
    a. Backups are chunked into Xmb (think 5 MB is default) chunks. The chunks aren't intuitively obvious that any other tool can be used for restoring. Expect that duplicity, Duplicati or DejaDup are required for a restore.

    b. These are full, then increment, increment, increment, increment, increment, increment, increment, increment, increment, increment, ... backups. I think they recommend doing a full backup monthly, so plan the time to backup once a month to be huge.
    https://help.ubuntu.com/community/DuplicityBackupHowto

    All backup tools have some downsides.

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •