New and need help for RAID

[DuckHook]

…My initial thought was that if I ran the system mirrored, then it would save me the time of ripping my media again if a drive failed

If a drive fails and does so in very specific ways (for HW reasons), then yes, the other HDD chugs away taking up the load. You would then replace the failed drive, re-activate mirroring and the replacement drive would shortly be identical to the surviving drive again.

But if you mistakenly delete half your files, then the mirrored drive would delete those files too. After all, that's what mirroring is supposed to do. If you need an old version of a file (say, an accounting database) from last month because you only discovered an error today, then without a backup, you are also out of luck. In my aforementioned scenario, you are hit by ransomware: a mirrored setup would just mirror the corruption. Both drives are now identically encrypted.

I guess I'm having trouble differentiating a mirror from a snapshot from a true backup.

Here's what I suggest:

1. Forget I ever mentioned "snapshots". That's just a high‑falutin concept that has no application to your needs. When you are ready to look into it, you will know what it is.
2. Likewise, mirroring, whether through mdadm, LVM or ZFS is not what you need.
3. Backups are essential. With your setup, a possible setup would be to use only one drive actively as your data storage. Use the second drive passively to duplicate the contents of the first drive, but only periodically at your instigation. That way, a mistake/menace in one drive does not automatically propagate to the other. This gives you basic redundancy without much trouble. But it's still not a backup.
4. For your real backups, here is what's important: you need to define a hierarchy of your data. Not all data is of like importance. Here's mine in ascending order of importance:
   
   1. Operating system: Worthless data. It can always simply be reinstalled. Ubuntu takes me less than 30 minutes to install from beginning to end. I don't ever bother backing my OS up.
   2. Convenience data: This is stuff that I already have source media for—CDs, DVDs, BDs. It's a pain to rip these to HDD again, but so what? It's just a matter of putting in the time. I rely on my redundant disks to give me enough protection here to feel safe.
   3. Important data: Family photos, home videos, etc. If I lost this stuff, Mrs DuckHook would kill me or I would consider doing so myself. Purely personal value, but high value. Many are okay backing this stuff up on some sort of cloud storage. After all, they have high personal value but are of little worth to bad guys. If you are okay with the metadata floating around out there on the cloud, then paying for an outside storage basket gives excellent peace of mind. Depending on how much of this you've got, it could get pricey though.
   4. Critical data: This is stuff like personal letters, contact databases, accounting databases, e-mail databases, tax/financial records, medical records etc. Stuff that's deeply personal and can be used to attack your assets or steal identity. And if lost, would constitute a true hardship or disastrous loss of records. This stuff is volatile (changes daily/weekly), highly sensitive and extremely valuable. Thankfully, these files also tend to be small. Mine fit on a USB stick. These can therefore be backed up cheaply and easily.

There it is. A quick and dirty backup regime. There are so many variations on the above and there's no one‑size‑fits‑all solution. Each person's needs are different and each needs to spin their own solution. It takes work and it takes thought, but when disaster strikes, you will pat yourself on the back that you were prepared. There are at least a dozen backup apps available for Linux. All Ubuntu flavours come with a backup utility called Déjà-Dup. For instructions, please see: https://www.linux.com/learn/total-sy...ecall-deja-dup

I'm not recommending this package. It is simply the one that is prepackaged so involves the least work and research.

BTW, those USB sticks are so cheap that I do two at once and keep one set off site at, say, my kid's place. A bank safety deposit box also does nicely.

Good luck and Happy Ubuntu-ing!
DH

[TheFu]

Nicely laid out DuckHook.

I have a plex server. Media files are huge and I can't afford to do proper, versioned, backups due to the cost of storage, but I can have a delayed mirror of the media files used by plex. For that, I use rsync a few times every week. Nothing on my Plex system uses RAID. This year, the OS HDD which contains the plex DB and metadata plus about 3TB of media failed. It was an inconvenience for a few days until a replacement HDD arrived. There wasn't any data loss besides a few TV recordings from that week. I think my plex system has about 36TB of storage connected, so about 18TB of media. There are 10 HDDs connected.

The OS and my personal files on the plex server are handled similar to what DuckHook does. I do not backup the OS either. I do backup the settings and OS data like the Plex metadata and DB using versioned backups. We'll ignore snapshots, but I use them as part of my backups to ensure no corruption happens of that information.

For things that need versioned backups, rsync isn't the best choice. Some more reading:

A simplified backup script: https://ubuntuforums.org/showthread.php?t=2436006 has more discussion.

Restoring backups: https://ubuntuforums.org/showthread....6#post13757986

rsnapshot: rsync + hardlinks for versioned backups (requires Linux file system like ext4)
https://popey.com/blog/2020/12/strai...ith-rsnapshot/
https://www.cyberciti.biz/faq/linux-...py-hard-links/

Until you've actually restored a system using your backups (and only your backups), you don't really have a solution. Plan on the first 5 attempts failing and tweaking the backups over time to capture some other useful information about the system to address other problems where a full restore isn't the best option.

[clctr13760]

1. 
   
   1. Convenience data: This is stuff that I already have source media for—CDs, DVDs, BDs. It's a pain to rip these to HDD again, but so what? It's just a matter of putting in the time. I rely on my redundant disks to give me enough protection here to feel safe.

This is the only data that will be on this system, and all important files are on my main PC with copies on spare flash drives.

TheFu,
Thank you for the links to more reading. I've never had a true backup solution, as the files I was worried about, just made it to a flash drive since they weren't very large. If something went wrong (which thankfully never has) I planned on copying them back after the problem was fixed.

[TheFu]

I've never had a true backup solution, as the files I was worried about, just made it to a flash drive since they weren't very large. If something went wrong (which thankfully never has) I planned on copying them back after the problem was fixed.

Everyone is like you, until there is a problem and 80% of your data is lost. That happened to me decades ago. That wasn't enough for me to get backup religion, however. I did get weekly-to-quarterly mirror religion. That saved me a few times, then I had gotten a few clients and was running some servers for them. In a professional shop, not having backups would be career ending, so I spend the time, the effort, did lots of testing of different solutions, but all the time had my mirroring answer before coming up with what I use today. You can skip all that effort by jumping to the decades proven solution immediately.

There is another option, of course. If you buy a $400 NAS device, most of them will have a way to backup to an external USB drive and to multiple cloudy services like S3-Glacier, Dropbox, and a number of other paid backup providers online. You'll have a local copy. You'll have the RAID box. You'll have some off-site backups for the really critical stuff which can also be encrypted. Trading money for time and expertise is a good solution for many people.

[rsteinmetz70112]

Mirroring will protect you somewhat from hardware failure. It will not protect you from yourself or from malicious software. Backups are essential to be able to recall data from one or more hopefully isolated backups which have more than one version.