Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: rsync: Reliably copying files from local file system to NAS

  1. #1
    Join Date
    Jun 2011
    Beans
    29

    rsync: Reliably copying files from local file system to NAS

    I run a virtual machine with Red Hat. On the virtual machine a node.js server is running and the data uploaded to the server is stored locally on the machine at /local/data.

    From time to time (e.g. once a week) I would like to move the data to a NAS. The NAS is mounted on the virtual machine. That means I would like to move the data from /local/data to /import/myNas/data

    /local/data contains folders, files and sqlite database files. The structure is as follows:

    Code:
    /local/data/folder1
             exception1.exception
             exception2.exception
             info.txt
             otherFile.txt
             /database
                 session1.db
                 session2.db
                 ....
                 session1000.db
        
    /local/data/folder2
             exception1.exception
             info.txt
             otherFile.txt
             /database
                 /session_198
                         session_198.00000034.db.tmp
                         session_198.00000035.db.tmp
                         ...
                         upload_fff344dbf979363020d1b874a8ee7bef.db
                 session1.db
                 session2.db
                 ....
                 session987.db
    I would like to copy all data to /import/myNas/data except for the folders in /database. So for example, /local/data/folder2/database/session_198 and all of its files should not be copied.

    How can I copy all these data to the NAS and when copying finished delete it from local storage? Very important is that the data is only deleted when the data is successfully copied (or moved), that means no corrupted or lost data. I don't know which the most reliable solution is (Python script, shell script, etc.)
    Last edited by Helveticus; August 12th, 2020 at 10:39 PM.

  2. #2
    Join Date
    Oct 2005
    Location
    Al Ain
    Beans
    10,051

    Re: rsync: Reliably copying files from local file system to NAS

    Something like this:
    rsync --exclude="*database*" --delete --max-delete=50 /local/data/ /import/myNas/data/

    The max-delete is intended to prevent a total catastrophe.

    Please see this also:
    https://www.aeronetworks.ca/2019/08/...de-simple.html
    Last edited by HermanAB; August 13th, 2020 at 05:35 AM.

  3. #3
    Join Date
    Nov 2007
    Location
    London, England
    Beans
    7,062
    Distro
    Xubuntu 20.04 Focal Fossa

    Re: rsync: Reliably copying files from local file system to NAS

    The --delete in rsync deletes files on the destination that don't have matching files in the source. This may not be what you are looking for.
    I think that to delete the source after a successful rsync would need a separate command. Something like this untested perhaps:
    Code:
    #!/bin/bash
    # tell the script to stop if any command fails - won't remove source files if rsync fails
    set -e 
    # copy the files
    rsync --exclude="*database*" --delete --max-delete=50 /local/data/ /import/myNas/data/
    # remove the source files
    rm -rf /local/data

  4. #4
    Join Date
    Oct 2005
    Location
    Al Ain
    Beans
    10,051

    Re: rsync: Reliably copying files from local file system to NAS

    Good one: 'set -e'
    That can save a whole lot of error checking in a script.

  5. #5
    Join Date
    Jun 2011
    Beans
    29

    Re: rsync: Reliably copying files from local file system to NAS

    Thank you very much to all of you.

    Does the set command get shipped with rsync or do I have to install some package?

    database should not be excluded but all folders inside
    database. How can I do this? The folders inside database start with "session", so for example database/session_234 or database/session_1234. But inside database there are also files starting with "session", such as database/session_234.db or database/session_1234.db. These files should not be excluded.

    Is it perhaps better to use --delete-source-files?

  6. #6
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    15,254
    Distro
    Kubuntu 20.04 Focal Fossa

    Re: rsync: Reliably copying files from local file system to NAS

    I much prefer to put includes and excludes in separate plain-text files and use
    Code:
    rsync -av --files-from=/usr/local/etc/rsync-includes --exclude-from=/usr/local/etc/rsync-excludes [etc.]
    You could then try populating rsync-includes with
    Code:
    [other filespecs you want to back up]
    database/session_*.db
    and rsync-excludes with
    Code:
    [other filespecs you want to exclude]
    database/session*
    You'll probably have to play around with these to get exactly the results you want.

    I don't know what kind of databases these are. I use PostgreSQL and MySQL. For both I run the "dump" command before running rsync, e.g., for PostgreSQL,
    Code:
    /usr/bin/pg_dumpall -U username > mypgdump
    That produces a plain-text file that can be used to reconstruct the databases if they go south. I use rsync to copy these files, not the databases themselves. I don't know if the database software you're using has an equivalent command.
    Last edited by SeijiSensei; August 13th, 2020 at 06:48 PM.
    If you ask for help, do not abandon your request. Please have the courtesy to check for responses and thank the people who helped you.

    Blog · Linode System Administration Guides · Android Apps for Ubuntu Users

  7. #7
    Join Date
    Jun 2011
    Beans
    29

    Re: rsync: Reliably copying files from local file system to NAS

    Thank you SeijiSensei. This helped me a lot. My script now looks as follows:

    Code:
    #!/bin/bash
    # chmod +x copy.sh
    # Execute script in root
    # tell the script to stop if any command fails - won't remove source files if rsync fails
    set -e
    # copy the files
    rsync --info=progress2 -r --include='database/session_*.db' --exclude  'database/session*' --remove-source-files /local/data/  /import/myNas/data
    This works perfectly. But I have just two questions:

    I found that --remove-source-files also removes not yet copied files when rsync stops. That means when I terminate rsync in-between copying files or when rsync stops due to a failure. How can I prevent this so that only files are delete in source which are successfully copied?

    Second, is it likely to get corrupted files from rsync? If a file is successfully copied but corrupted and gets removed by --remove-source-files, that would be very bad. Is there a way to guarantee that only non-corrupted files are deleted in the source directory?
    Last edited by Helveticus; August 13th, 2020 at 07:47 PM.

  8. #8
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    15,254
    Distro
    Kubuntu 20.04 Focal Fossa

    Re: rsync: Reliably copying files from local file system to NAS

    I've never used --remove-source-files. If I want to delete files on the source, I use a script on that machine. Most of the time I just leave the source files alone.

    As for file checking, see https://www.oreilly.com/library/view...9/ch01s16.html

    Basically it suggests running rsync twice, once to transfer the files, and again with the "-n" or "--dry-run" switch after the initial transfer is complete.
    Last edited by SeijiSensei; August 13th, 2020 at 08:47 PM.
    If you ask for help, do not abandon your request. Please have the courtesy to check for responses and thank the people who helped you.

    Blog · Linode System Administration Guides · Android Apps for Ubuntu Users

  9. #9
    Join Date
    Jun 2011
    Beans
    29

    Re: rsync: Reliably copying files from local file system to NAS

    Thanks for the link. That makes sense, I will give it a try.

    My script now looks as follows:

    Code:
    #!/bin/bash
    # chmod +x copy.sh
    # Execute script in root
    # tell the script to stop if any command fails - won't remove source files if rsync fails
    set -e
    # copy the files
    rsync --info=progress2 -r --include='database/session_*.db' --exclude 'database/session*' /local/data/ /import/myNas/data
    rsync --info=progress2 -r --include='database/session_*.db' --exclude 'database/session*' --checksum --remove-source-files /local/data/ /import/myNas/data
    The problem now is that while rsync is running new files are written to /local/data. I would like that rsync takes a snapshots of the list of files when it runs the first time and then only copies these files. In the second run rsync should then also only run on these files from the snapshot. Is this possible?
    Last edited by Helveticus; August 13th, 2020 at 10:15 PM.

  10. #10
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    15,254
    Distro
    Kubuntu 20.04 Focal Fossa

    Re: rsync: Reliably copying files from local file system to NAS

    What if you reverse the comparison in the second step comparing the files on the NAS to their origins?
    If you ask for help, do not abandon your request. Please have the courtesy to check for responses and thank the people who helped you.

    Blog · Linode System Administration Guides · Android Apps for Ubuntu Users

Page 1 of 2 12 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •