Results 1 to 7 of 7

Thread: rsync does not send files correctly

  1. #1
    Join Date
    Aug 2012
    Beans
    3

    rsync does not send files correctly

    Hello all. I'm a relative *nix noob, but to save money at work I decided to put in an Ubuntu server as a fileserver so that we could use it as a web server as well. Things have been going all right, except for that I discovered this weird issue today:

    I run rsync nightly to a remote server. The update is one-directional--I don't need to pull anything back over, I just dump the local server's files to a remote server. Cron runs nightly (as root) and runs the backup sh script.

    Today, I noticed that files weren't being backed up, and hadn't been for a little while, so I ran the backup script by itself, using the command:

    Code:
    sudo sh /scripts/backup
    It did the weirdest thing... it listed all the new and changed files... and then it was done. Now, the connection between the two servers is a T1, so I knew the 1.2GB ISO file I stuck on the server today wasn't tranferred in a few seconds. So I switched on the verbose switch on rsync and re-ran. Here's the script (pared down for the sake of simplicity):


    Code:
    #Establish time
    echo "Job start time:"
    date
    
    echo "----------------------------"
    date
    echo "Backing up Working share..."
    rsync -arv /data/Working/    /mnt/abc/Working/
    
    echo ""
    date
    echo "Backing up Accounting share..."
    rsync -arv /data/Accounting/ /mnt/abc/Accounting/
    
    #Establish finish time
    echo "--------------------------"
    echo "Job end time:"
    date
    
    #Complete
    echo "Back up complete."
    The "mnt" directory is a group of remote server directory mounted via mount.cifs. I am able to browse these directories, enumerate /open/copy files, etc. The connection appears just fine.

    So then I run the backup script, and I get output like this:

    Code:
    Thu Sep 27 12:30:44 EDT 2012
    Backing up IT share...
    sending incremental file list
    Software/
    Software/OS and Images/
    Software/OS and Images/ubcd511.iso
    Software/Productivity/
    Software/Tools & Utilities/
    Software/Tools & Utilities/System Information Utilities/
    Software/Tools & Utilities/System Information Utilities/SysinternalsSuite.zip
    
    sent 13134695137 bytes  received 2206 bytes  81836120.52 bytes/sec
    total size is 44466830195  speedup is 3.39
    81 MEGABYTES PER SECOND! !!

    Of course, I knew that wasn't right, so I checked the destination and sure enough none of the changes were actually applied.

    Anybody have any idea what's going on?

    Dan

  2. #2
    Join Date
    Jun 2008
    Location
    Tennessee
    Beans
    3,415

    Re: rsync does not send files correctly

    Are you quite sure the destination point is actually mounted? What does the output of "mount" (with no arguments) tell you?

    If it weren't, rsync would just be copying to the local disk, which would explain the speed.

    If the script had run at some point when the remote system wasn't mounted, and data had been copied into the mount points, subsequent attempts to mount the remote server may have failed since the mount point wasn't empty.

    That's my theory.

  3. #3
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,106
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: rsync does not send files correctly

    Personally, I wouldn't rely on mounts over a T1. I would be using passwordless SSH with keys and modify the script as such.

    Code:
    #Establish time
    if ping -c 1 remote_ip
    then
      echo "Job start time:"
      date
    
      echo "----------------------------"
      date
      echo "Backing up Working share..."
      rsync -arv /data/Working/    root@remote_ip:/path/to/files/Working/
    
      echo ""
      date
      echo "Backing up Accounting share..."
      rsync -arv /data/Accounting/ root@remote_ip:/path/to/files/Accounting/
    
      #Establish finish time
      echo "--------------------------"
      echo "Job end time:"
      date
    
      #Complete
      echo "Back up complete."
    else
      echo "=====The remote server is down====="
      echo "setup email alert here" 
    fi
    This way you know your files are going to the correct spot. Also, it will check before it runs to ensure that the remote server is up. If it's not, it should send you an email so you can troubleshoot the problem.

  4. #4
    Join Date
    Aug 2012
    Beans
    3

    Re: rsync does not send files correctly

    Quote Originally Posted by lykwydchykyn View Post
    Are you quite sure the destination point is actually mounted? What does the output of "mount" (with no arguments) tell you?

    If it weren't, rsync would just be copying to the local disk, which would explain the speed.

    If the script had run at some point when the remote system wasn't mounted, and data had been copied into the mount points, subsequent attempts to mount the remote server may have failed since the mount point wasn't empty.

    That's my theory.
    Thanks for the replies. As always happens, I solved my problem after posting this. I restarted the server (it hadn't restarted in a couple weeks) and everything started working fine.

    What confuses me is that I could traverse the remote directories--which is completely different from the behavior I typically see if the remote server disconnect. Hmm.

    What is the rationale for using SSH rather than mount points?

    Thank you!

    Dan

    EDIT: I'm backing up to a Windows file server, if that makes a difference.
    Last edited by dipique; September 28th, 2012 at 01:01 PM.

  5. #5
    Join Date
    Jul 2010
    Location
    Michigan, USA
    Beans
    2,106
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: rsync does not send files correctly

    Quote Originally Posted by dipique View Post
    Thanks for the replies. As always happens, I solved my problem after posting this. I restarted the server (it hadn't restarted in a couple weeks) and everything started working fine.

    What confuses me is that I could traverse the remote directories--which is completely different from the behavior I typically see if the remote server disconnect. Hmm.

    What is the rationale for using SSH rather than mount points?

    Thank you!

    Dan
    You have to create a path for a mount point, in your case /mnt/abc/Working/ and /mnt/abc/Accouting/. If your remote share is not available, or isn't mounted, that path is still available on the original machine. You could believe that your backup is successful, but you're only backing up to the original machine.

    The way I posted would ensure the remote host is available and then would backup to it if it was. Also, you would receive an email if it was not available. None of those factors are being prevented in your current implementation.

  6. #6
    Join Date
    Aug 2012
    Beans
    3

    Re: rsync does not send files correctly

    Quote Originally Posted by rubylaser View Post
    You have to create a path for a mount point, in your case /mnt/abc/Working/ and /mnt/abc/Accouting/. If your remote share is not available, or isn't mounted, that path is still available on the original machine. You could believe that your backup is successful, but you're only backing up to the original machine.

    The way I posted would ensure the remote host is available and then would backup to it if it was. Also, you would receive an email if it was not available. None of those factors are being prevented in your current implementation.
    I see. Could you give me some guidance in terms of how to set that up on a Windows server? Or point me in the right direction?

    Dan

  7. #7
    Join Date
    Jun 2008
    Location
    Tennessee
    Beans
    3,415

    Re: rsync does not send files correctly

    Quote Originally Posted by rubylaser View Post
    Personally, I wouldn't rely on mounts over a T1. I would be using passwordless SSH with keys and modify the script as such.
    I use this method as well; it seems a more direct approach than mounting a drive, which has potential issues as I highlighted.

    No idea how to set that up on Windows, other than that you need to install an SSH server with SCP and SFTP support enabled. A quick google shows that there are a number to choose from.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •