Results 1 to 7 of 7

Thread: rsync permission error but files backed up

  1. #1
    Join Date
    Aug 2012
    Beans
    90

    rsync permission error but files backed up

    I'm using rsync in a bash script to backup files from a Ubuntu 16.04LTS server to a local Ubuntu 18.04LTS server.. The rsync statement below
    Code:
    rsync  -azhq --stats --exclude=.* --log-file=backup_myfileserver.log /home/myfileserver /mnt_remote_backup/${backup_destination}
    I'm getting errors in the log for some files, but all files, even the ones with the error message seem to have been transferred okay. Is this normal? Is there anything I can do to remove the error messages
    Thanks
    Code:
    2019/05/15 11:48:41 [3155] building file list
    2019/05/15 11:48:41 [3155] rsync: chgrp "/mnt_remote_backup/2019-05-15/myfileserver" failed: Permission denied (13)
    2019/05/15 11:48:42 [3155] rsync: chgrp "/mnt_remote_backup/2019-05-15/myfileserver/personal" failed: Permission denied (13)
    2019/05/15 11:48:42 [3155] rsync: chgrp "/mnt_remote_backup/2019-05-15/myfileserver/personal/personal" failed: Permission denied (13)
    2019/05/15 11:48:42 [3155] rsync: chgrp "/mnt_remote_backup/2019-05-15/myfileserver/personal/photography" failed: Permission denied (13)
    


  2. #2
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    16,278
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: rsync permission error but files backed up

    If the userid isn't a member of the group and the groups don't exist on both systems, yes.
    rsync is only an effective backup tool if the files are all owned by 1 userid and that userid is the one running the rsync command.

    Of course, the storage needs to be Linux on a file system, not NTFS nor FAT-whatever. Those don't support Unix-style permissions through the available drivers.

    BTW, why not use rsync -avz local-source/ userid@remote-IP:/target/directory/ for the command to see what the issues are a little better, then once things are working, you can remove the -v and put in the -q.

    From a security standpoint, backups should be pulled by the backup server using non-remote mount protocols. rsync fits that nicely. By using network mounts (it appears you are) and mounting on the client machine, then malware can totally destroy all the backups. With the backup server pulling data and not allowing the client machine(s) direct access to that storage, you've just protected your backups from crypto-locker-type attacks.

    Is there a reason not to use a real backup tool that handles versioning for you? They are crazy efficient these days and the command looks pretty much like an rsync command, but provides so much more capability for almost zero extra effort and less than 10-15% more storage. Just a thought.

    If you are doing HOME directory backups, not using root (or a root-equiv) userid is fine. If you are doing backups for systems, then using root really is the only method that can also be secured.

    I'm happy to explain anything above if it isn't clear. Excellent, versioned, backups are really easy these days.

  3. #3
    Join Date
    Aug 2012
    Beans
    90

    Re: rsync permission error but files backed up

    Thanks for the detailed answer. I have a few questions
    Code:
    If the userid isn't a member of the group and the groups don't exist on both systems, yes. rsync is only an effective backup tool if the files are all owned by 1 userid and that userid is the one running the rsync command.

    I think I have a problem with one user/userid but I can fix that. But for now are my backups okay in spite of the error messages
    Code:
    Of course, the storage needs to be Linux on a file system, not NTFS nor FAT-whatever. Those don't support Unix-style permissions through the available drivers.

    Yes it's on a Linux file system - ext4
    Code:
    BTW, why not use rsync -avz local-source/ userid@remote-IP:/target/directory/
    for the command to see what the issues are a little better, then once things are working, you can remove the -v and put in the -q.
    I was using the v but the output was overwhelming so I tried the q to see if it was easier to understand
    Code:
    From a security standpoint, backups should be pulled by the backup server using non-remote mount protocols.  rsync fits that nicely.  By using network mounts (it appears you are) and mounting on the client machine, then malware can totally destroy all the backups.  With the backup server pulling data and not allowing the client machine(s) direct access to that storage, you've just protected your backups from crypto-locker-type attacks.
    I agree and will change it
    Code:
    Is there a reason not to use a real backup tool that handles versioning for you?  They are crazy efficient these days and the command looks pretty much like an rsync command, but provides so much more capability for almost zero extra effort and less than 10-15% more storage.  Just a thought. If you are doing HOME directory backups, not using root (or a root-equiv) userid is fine.  If you are doing backups for systems, then using root really is the only method that can also be secured.
    Its a hobby site and I'm using bash and rsync as a learning experience.

    Thanks again for your help

    Frank

  4. #4
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    13,946
    Distro
    Kubuntu 19.04 Disco Dingo

    Re: rsync permission error but files backed up

    One solution is to run the command as root with sudo. I write all my backups as root so I don't need to worry about permissions.
    If you ask for help, do not abandon your request. Please have the courtesy to check for responses and thank the people who helped you.

    Blog · Linode System Administration Guides · Android Apps for Ubuntu Users

  5. #5
    Join Date
    Aug 2012
    Beans
    90

    Re: rsync permission error but files backed up

    Thanks for the suggestion.
    I decided to take this opportunity to straighten out the owner/groups permissions for all the files. That seemed to get rid of the errors.
    Frank

  6. #6
    Join Date
    Mar 2010
    Location
    Squidbilly-Land
    Beans
    16,278
    Distro
    Ubuntu Mate 16.04 Xenial Xerus

    Re: rsync permission error but files backed up

    In general, I really appreciate 'code tags', but not when quoting others. Makes reading the paragraph hard when it doesn't wrap. "quote tags" would be great above.

    Its a hobby site and I'm using bash and rsync as a learning experience.
    Good enough. We all have to start somewhere and mastering rsync **is** an excellent use of your time.

    I agree with SeijiSensei about using root/sudo for backups to retain permissions. For local backups to local storage, this is the easy method. For network-based backups, there are some added complexities in using sudo/root between Ubuntu systems.

    Anyways, the rdiff-backup tool uses librsync and provides versioning. The most recent backup is just like an rsync mirror, plus it keeps track of file permissions changes over time, unlike rsync. rsync only knows the last permissions for a file/owner/group. rdiff-backup knows whenever they change and stores that data in the versioning. For backups of files all owned by the same userid, like just a single HOME directory, this isn't too important. But for whole system backups, retaining the correct permissions is absolutely critical.

    Code:
    $ sudo rdiff-backup --exclude-special-files --exclude=.*  /home/myfileserver /mnt_remote_backup/${backup_destination}
    should work. I'd need to look at the --exclude, since globbing in rsync/rdiff-backup don't work like normal file system globbing does. Both rsync and rdiff-backup follow the same rules, however. I wouldn't exclude all .files like that, since Unix uses them to store configurations for many, many, different programs. These are usually stored in $HOME, but sometimes they are stored elsewhere - like .gitignore, .vimrc, and other $PWD specific tools. If you want to exclude .cache/ directories, --exclude=**/.cache is how I'd do it. Of course, there are 50 different ways to handle this. Whatever works is good.

  7. #7
    Join Date
    Aug 2012
    Beans
    90

    Re: rsync permission error but files backed up

    I'll check out the rdiff-backup
    Thanks
    Frank

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •