Page 2 of 2 FirstFirst 12
Results 11 to 14 of 14

Thread: Rsync problems on huge collection of local files

  1. #11
    Join Date
    Aug 2008
    Location
    Waihi Beach, New Zealand
    Beans
    108
    Distro
    Ubuntu 8.04 Hardy Heron

    Re: Rsync problems on huge collection of local files

    Hi VidTek. Thanks for suggestion, but had looked at that fsarchiver earlier. FSArchiver is a system tool that allows you to save the contents of a file-system to a compressed archive file.
    What I need is a faster (file) replacement for rsync and/or the rm -r-d command. The latter has been achieved with the perl script.
    Last night I set task of (caja) copy from ntfs partition to ext4 partition. The job hung with permissions issue and estimated completion time of 17 hours.
    I've decided to go further with the perl script to include (recursive) dircopy, dirmove and dirdelete. Perl can preserve file attributes across linux partitions, and whatever attributes are on ntfs. Once Ive got the script working, will do an a/b compare to rsync and GNU mv.

  2. #12
    Join Date
    Jan 2011
    Location
    Bournemouth UK
    Beans
    311
    Distro
    Kubuntu 18.04 Bionic Beaver

    Re: Rsync problems on huge collection of local files

    AG- Glad you are progressing with the issues that forced me to just give up and buy a couple of extra drives.

    Funny thing is I also use 2 x 4tb Seagate USB Expansion 2 1/2" drives too as my main data and backup.

    I just ripped them both out of their cases and stuck one inside my tower along with the 5 tb. WD drive 3 1/2" I use for my multimedia setup.

    Like I said I whack both Seagates and WD drives into the cloning dock once a month. It takes about 6 hrs to clone the 4tb drives in the dock, and

    about 4 1/2 hrs for the WD drives.
    Asus Z270 i7 16gb ram 8.25tb GT960 TSB6205 Quad tuner 64-bit Kubuntu 18.04 Bionic & win 10 Be/FE mythtv 0.29
    Laptop Samsung NP R580 i5 nvidia linux ultimate & win 10 Homerun dual netwk tuner 55¨ Samsung ES8000

  3. #13
    Join Date
    Aug 2008
    Location
    Waihi Beach, New Zealand
    Beans
    108
    Distro
    Ubuntu 8.04 Hardy Heron

    Re: Rsync problems on huge collection of local files

    Hi vidtek again.....Im putting the finishing touches on perl script would you like to try it out when done? do you know how to run perl scripts?

  4. #14
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: Rsync problems on huge collection of local files

    Here's my media rsync script to backup a 4TB data disk to another 4TB data disk.
    Code:
    $ EXCLUDES="--exclude .Trash-1000 --exclude lost+found --exclude ZCS-2012"
    $ ionice rsync -av --stats --progress $EXCLUDES --delete-before /misc/D5/ /misc/b-D5/
    This is just 1 of the 20TB of storage and only a media/data file backup.

    For OS and HOME directory backups, I want versioning.
    Basically, that is
    Code:
    sudo rdiff-backup    --exclude-special-files   /usr/local  /root  /home  /opt  /etc  /var/www /var/lib/libvirt /Backups/$HOSTNAME/
    The first run takes whatever an rsync takes for time.
    All runs after that only transmit changed/new files or changed/new metadata about files. For most systems, it is 2-4 minutes.

    Yes, some pre-backup data is gathered to make restoring the system to the same place it was before possible. For example, the list of manually installed packages. I use a perl script for my backups,
    Code:
    `$aptmark showmanual | tee $local_backup/apt-mark.manual`;
    ######[ to restore pkgs ]#######
    ### sudo apt-mark auto $(cat pkgs_auto.lst)
    ### sudo apt-mark manual $(cat pkgs_manual.lst)
    ### sudo apt-get -u dselect-upgrade
    I also backup ruby gems, cpanm module lists, etc.
    Code:
    # Daily dump of current ruby gems to $HOME/backup - 
     `$gem list --local > $local_backup/gem.list`;
    # Daily dump of current CPAN modules to $HOME/backup - cpan -l
     `$cpan -l > $local_backup/cpan.list ` if ( -e "/usr/local/lib/site_perl");
    Restore process is:
    1. Fresh install of same/similar Ubuntu flavor to old or new system
    2. Restore all the backed-up data, except /etc/ files. Selectively restore those as needed - /etc/apt/, /etc/ufw/, /etc/iptables/, /etc/hosts, /etc/netplan/ (or network), and a few other files that I manually changed over the days, months, years, prior.
    3. Take the list of manually installed packages and feed those into apt-mark and apt-get to install. It is important to have both data and settings already in the expected locations BEFORE installing the packages so those settings are used.

    This method works well for me and others. 15 min for a fresh OS install, using the normal, stock, installer. Change the storage however you like, diff file systems, diff storage amounts, LVM or not, whatever. 15 min to restore the backup data - pick the version you want from last night, 13 days ago, 79 days ago, whenever. I usually have 90 - 180 days of backup versions per system. That's 30 minutes to a restored system that "feels" just like the prior system. Only huge data like videos and audio files might not be restored at this point. Best of all, the backups don't grow by the first backup size. They grow only as much as the diff-gz for each new file requires. 60 days of versioned backups usually take 10-20% more storage than the original mirror size. A server with 10G in a backup is 13G for 90 days of versioned backups. Why wouldn't people choose that?

    Been using this method for about 9 yrs now. About once a year, I'll either do something stupid or have a HDD fail on one of the systems and get to use the backups to restore.

    Some systems don't hold any data, so the backups only contain configs/settings for that server. My email gateway server uses less than 50MB for 180 days of versioned backups. Heck, there is little reason I shouldn't keep every version until it is time to move to a fresh install of that gateway. Backups are tiny.

    All backup systems/solutions are an exercise in trade-offs.
    Last edited by TheFu; 4 Weeks Ago at 02:52 AM.

Page 2 of 2 FirstFirst 12

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •