Page 1 of 3 123 LastLast
Results 1 to 10 of 27

Thread: Slow write speed when backing up internal HDD to external USB3.0 HDD

  1. #1
    Join Date
    Nov 2017
    Beans
    101

    Slow write speed when backing up internal HDD to external USB3.0 HDD

    I performed a backup of a HDD in my server but it seemed to take a very long time

    It took almost 2.5 hours to backup around 650GB, which seems a long time to me

    The hardware is as follows:




    I used Free File Sync in Mirror mode (on Ubuntu 20.04) to perform the backup from the source HDD to the destination HDD.

    This was the first backup I performed with the external HDD, it had just been formatted as exFAT with a single partition.

    I have some other internal 4TB HDD to backup, one of which is nearly full.

    At this speed it would take over 15 hours to backup that 4TB HDD !

    Surely this can't be right ?
    Last edited by freeflyjohn; July 9th, 2021 at 07:20 PM.

  2. #2
    Join Date
    Nov 2017
    Beans
    101

    Re: Slow write speed when backing up internal HDD to external USB3.0 HDD

    I performed a benchmark on the external HDD (connected via USB3.0) using the Disks app in Ubuntu, the results are shown below...

    Attached Images Attached Images

  3. #3
    Join Date
    Jun 2019
    Location
    Dirndl-land
    Beans
    851
    Distro
    Lubuntu 20.04 Focal Fossa

    Re: Slow write speed when backing up internal HDD to external USB3.0 HDD

    Why are you copying ext4 to exFAT? Sounds like a bad idea to me.

  4. #4
    Join Date
    Nov 2017
    Beans
    101

    Re: Slow write speed when backing up internal HDD to external USB3.0 HDD

    ext4 is the default format for Ubuntu Linux

    The reason for the external HDD being exFAT is for compatibility, so I can read the HDD using a Mac or Windows machine.

    Could this be the reason for the slow write speeds ?

    If so, what format should the external HDD be ?

    Why is copying from ext4 to exFAT a bad idea out of interest ?
    Last edited by freeflyjohn; July 9th, 2021 at 08:37 PM.

  5. #5
    Join Date
    Jun 2019
    Location
    Dirndl-land
    Beans
    851
    Distro
    Lubuntu 20.04 Focal Fossa

    Re: Slow write speed when backing up internal HDD to external USB3.0 HDD

    Quote Originally Posted by freeflyjohn View Post
    Why is copying from ext4 to exFAT a bad idea out of interest ?
    AFAIK, FAT or exFAT only allow user space operations, which is suboptimal for UNIX/Linux.

    You'd probably be better off with NTFS.

  6. #6
    Join Date
    Sep 2009
    Location
    Pennsylvania
    Beans
    3,547
    Distro
    Xubuntu

    Re: Slow write speed when backing up internal HDD to external USB3.0 HDD

    NTFS will be your best choice if you cannot use ext4.

  7. #7
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Slow write speed when backing up internal HDD to external USB3.0 HDD

    There are a few rules of thumb I follow when moving data around.
    • Use file systems supported by the kernel, not FUSE. That usually means don't use anything with NTFS or *FAT* in the name as a file system.
    • USB3 and newer connections are fine, assuming the device is externally powered. Devices powered only through the USB port will have slow performance. Some non-spinning HDDs are exceptions - SSDs - for example.
    • Don't use a GUI. GUIs spend 90% of their time updating the GUI rather than moving bits around. If just a few files, this doesn't matter. For 20+ files, use rsync.
    • When using rsync, local copies are less efficient to network copies. rsync treats local storage different than network storage. The rsync manpage explains more.
    • Lots of tiny files are less efficient than fewer, larger, files when rsync is used. If creating a backup, use a real backup tool, not rsync. rdiff-backup is faster than rsync AFTER the initial run completes. This is because checksums are pre-calculated and stored for all files already and don't need to be re-calculated on future runs on the target storage. In short, rdiff-backup does haven't the work in comparing file blocks when compared to rsync, so it is faster.


    I recall reading that exFAT with kernel drivers was coming about a year ago, but this was a Microsoft thing, so I knew not to hold my breath. I'm still waiting for "Longhorn" OS to be released by MSFT. Maybe next year?

    If using either exFAT or NTFS, be certain to include the performance mount options. Don't just point-n-click to gain access. My notes about that:
    • NTFS performance - async,big_writes
    • exFAT performance - dirsync in kernel-based exfat mount code.


    I can confirm that big_writes can improve NTFS performance about 30%. The exFAT option only works if you use the kernel-mode file system, not the FUSE version. I've never, ever, used exFAT, so don't know anything about it. I use f2fs for flash storage whenever possible. and FAT32 for cameras and other devices that don't support f2fs. The performance specs for f2fs show it beating ext4 in many tests.

  8. #8
    Join Date
    Nov 2017
    Beans
    101

    Re: Slow write speed when backing up internal HDD to external USB3.0 HDD

    Thanks all

    @TheFu: I have reformatted the external USB HDD as ext4 and mounted in /etc/fstab using...

    Code:
    UUID=42d3df00-6b9c-4f9f-b46d-cf3eb1c6a5df media/backupsda ext4 noatime,nodiratime,x-gvfs-name=BackupSDA 0 2
    I have installed rdiff-backup and its currently performing the same backup I originally tried with Free File Sync

    It's a shame that theres no progress bar, I have no idea how far through the backup it is.

    Hopefully it will be quicker than 2.5 hours, although I dont know how I will be able to check the time taken to backup once its complete ?

    The HDDs in my server that I want to backup mainly contain personal data such as media (pictures, video, documents etc.)

    I don't think theres many system files on these HDDS, other than svn, nextcloud and plex.

    I was reading that rdiff-backup was not the best tool for media files and that rsync would be better ?

    I only plan to do manual backups of the HDDs in the server, I just want some form of backup in case one of these HDDs fail.

  9. #9
    Join Date
    Nov 2017
    Beans
    101

    Re: Slow write speed when backing up internal HDD to external USB3.0 HDD

    It took just as long using rdiff-backup (with the external HDD formatted as ext4) as it did using Free File Sync (with the external HDD formatted as exFAT) - around 2.5 hours to backup 750GB

  10. #10
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    Hidden!
    Distro
    Ubuntu

    Re: Slow write speed when backing up internal HDD to external USB3.0 HDD

    I'm surprised that fstab entry worked. I don't think it did because it is incorrect - missing a '/' before media. Also, putting manually mounted storage under /media is a bad idea, but you can learn that on your own.

    rdiff-backup is for files that change all the time, not so much for static files like videos and audio files. Photos should be fine because they aren't huge and you might add some exif data later ... or even just rotate some images.
    Plex stuff can become huge if you've enabled "optimization". That can easily eat 200G under "/var/lib/plexmediaserver/Library/Application Support/Plex Media Server" I'm not joking about that location. The Plex Guys love to waste 45 characters for no good reason. /var/lib/plex/ would be just as useful.

    I was reading that rdiff-backup was not the best tool for media files and that rsync would be better ?
    I may have written that. I use rsync for media files because I can't keep 180 copies of it - practical storage problem. OTOH, I can easily keep 180 versions of my "what I need to recreate MY system" files.

    Doing manual backups is a terrible idea. At least setup weekly automatic backups at the minimum. Human nature gets in the way of us doing anything manual. I know. I was the same 20 yrs ago - manual backups - but over time I got lazy. Best to just realize that you'll be lazy now and setup automatic backups. If you connect and disconnect the storage during the week, that can be scripted as a check and a reminder sent to your twitter feed or email or voice mail, if needed, to connect the storage before the scheduled time.

    rdiff-backup creates metadata files which contain all sorts of information. I've scripted a little to pull that data out and do a little math.
    For example, my backup script dumps this:
    Code:
    INFO: Backing up .... --include /root --include /etc --include /var/www --include /usr/local --include /home    to 
                   backups@romulus::/Backups/spam3/
    INFO: Removing backups older than 180D.
    INFO: Backup Start: Sat Jul 10 01:30:01 EDT 2021
    INFO: Backup End: Sat Jul 10 01:30:09 EDT 2021
    INFO: Backup Job Complete.
    during the run. That system is an email gateway server, so it doesn't actually have any data, just configuration settings. That's why the backup was 8 seconds.

    Under each machine's backup directory, there's a rdiff-backup-data/ directory. In there is a session_statistics.2021* file for the date which contains all sorts of information to be slurped into a summary report. For example:
    Code:
    === Time for Backups to regulus === 
    StartTime 1625894103.00 (Sat Jul 10 01:15:03 2021)
    EndTime 1625894173.84 (Sat Jul 10 01:16:13 2021)
    ElapsedTime 70.84 (1 minute 10.84 seconds)
    TotalDestinationSizeChange 263461 (257 KB)
    That's just using egrep. Obviously, regulus has been backed up previously. Every week, on Sunday mornings, I create a report that contains the backup increments for each system. Staying with regulus .... the filename is inc-sizes-regulus-2021-07-04
    Code:
            Time                       Size        Cumulative size
    -----------------------------------------------------------------------------
    Sun Jul  4 01:15:04 2021         6.11 GB           6.11 GB   (current mirror)
    Sat Jul  3 01:15:04 2021         2.37 MB           6.11 GB
    Fri Jul  2 01:15:04 2021         3.06 MB           6.11 GB
    Thu Jul  1 01:15:03 2021         2.60 MB           6.11 GB
    Wed Jun 30 01:15:04 2021         2.70 MB           6.12 GB
    Tue Jun 29 01:15:04 2021          172 MB           6.28 GB
    ...
    Thu Apr  8 01:15:03 2021         8.26 MB           9.33 GB
    Wed Apr  7 01:15:04 2021         2.42 MB           9.33 GB
    Tue Apr  6 01:15:04 2021         2.92 MB           9.33 GB
    So for 90 days of versioned backups, which holds about 6.11G today, all those backups use just 9.33G of storage. That is sufficient information for me to blow away regulus, do a fresh install, restore stuff from the backup, and have the desktop "feel" like my desktop with all the data, programs, settings, and configuration. Recall that I don't keep media files on my desktop. Those are stored on the network under /d/ somewhere and NFS mounted for use automatically, on-demand. So, right now, regulus doesn't have any of that storage mounted:
    Code:
    $ dft
    Filesystem                      Type  Size  Used Avail Use% Mounted on
    /dev/mapper/vgubuntu--mate-root ext4   19G   15G  3.6G  81% /
    /dev/mapper/vgubuntu--mate-home ext4   12G  6.4G  4.9G  57% /home
    /dev/vda1                       vfat  511M  7.1M  504M   2% /boot/efi
    but if I just access it with either an ls or cd command, it will be mounted.
    Code:
    $ dft
    Filesystem                      Type  Size  Used Avail Use% Mounted on
    /dev/mapper/vgubuntu--mate-root ext4   19G   15G  3.6G  81% /
    /dev/mapper/vgubuntu--mate-home ext4   12G  6.4G  4.9G  57% /home
    /dev/vda1                       vfat  511M  7.1M  504M   2% /boot/efi
    istar:/d/D1                     nfs4  3.5T  3.5T   12G 100% /d/D1
    If I don't do anything more with it, in a few minutes, the storage in /d/D1 will automatically umount. If I access D3 ... then it gets mounted:
    Code:
    $ dft
    Filesystem                      Type  Size  Used Avail Use% Mounted on
    /dev/mapper/vgubuntu--mate-root ext4   19G   15G  3.6G  81% /
    /dev/mapper/vgubuntu--mate-home ext4   12G  6.4G  4.9G  57% /home
    /dev/vda1                       vfat  511M  7.1M  504M   2% /boot/efi
    istar:/d/D1                     nfs4  3.5T  3.5T   12G 100% /d/D1
    istar:/d/D3                     nfs4  3.6T  3.6T   13G 100% /d/D3
    istar is my jellyfin/plex/calibre/mpd server with over 32TB of storage. The primary media storage is available to my nextcloud instance (read-only access) and a few desktops/laptops using NFSv4.

    If you have rsync data, it is best to keep that outside the places that rdiff-backup grabs. I keep my media server files under /d/D[1-9] and block other tools from backing anything under /d/ up, except rsync. The rsync jobs run every other day. The rdiff-backup jobs run nightly.

Page 1 of 3 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •