Page 1 of 2 12 LastLast
Results 1 to 10 of 18

Thread: rsync and progress in log files

  1. #1
    Join Date
    Feb 2008
    Location
    Chicago, IL
    Beans
    112
    Distro
    Ubuntu 12.10 Quantal Quetzal

    rsync and progress in log files

    I have setup a rsync cron job to sync some data from my server to my local box, but it is alot of data. Since all this is being done via cron jobs, the process runs in the background, but I would like to be able to track its progress.

    How do I track the progress of this cron job while rsync syncs in the background?

    EDIT
    I have already implemented -v option and -P option, but it only shows when the file was completed in the log file. The -P option only shows up when you run it at the terminal, not as a cron job. I need a way to peering into the process itself if the -v -P isn't giving me up to the second information of the remaining bytes of the file(s) being transferred in the log file.
    Last edited by diskmaster23; February 12th, 2012 at 04:27 PM. Reason: Clarification

  2. #2
    Join Date
    Feb 2012
    Beans
    18

    Re: rsync and progress in log files

    What a wonderfully liberal forum.

    Certainly no Stack Exchange (from where I was banned, apparently just for wanting to get involved and help...)

    Just search on the web for how to schedule rsync jobs on linux. You will likely want email notification as well.

    Any solution will do, it's not specific to ubuntu.

    And before getting narky that I did not actually answer, take a breath and appreciate my answer for it will likely have the magic effect of causing another to come along and go one step further and actually provide you the direct answer

  3. #3
    Join Date
    Feb 2009
    Location
    Dallas, TX
    Beans
    6,590
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: rsync and progress in log files

    A couple of ideas:
    • use rsync with the verbose option, and redirect its output to a log that you can check.
    • get your script inside a deattached virtual terminal (using the program 'screen'), so you can attached it any time you need to see what is going on.

    Hope it helps, and tell us if you need more details on those ideas.
    Regards.
    Last edited by papibe; February 12th, 2012 at 08:51 AM. Reason: spelling

  4. #4
    Join Date
    Feb 2008
    Location
    Chicago, IL
    Beans
    112
    Distro
    Ubuntu 12.10 Quantal Quetzal

    Re: rsync and progress in log files

    Quote Originally Posted by sleepyhollows View Post
    What a wonderfully liberal forum.

    Certainly no Stack Exchange (from where I was banned, apparently just for wanting to get involved and help...)

    Just search on the web for how to schedule rsync jobs on linux. You will likely want email notification as well.

    Any solution will do, it's not specific to ubuntu.

    And before getting narky that I did not actually answer, take a breath and appreciate my answer for it will likely have the magic effect of causing another to come along and go one step further and actually provide you the direct answer
    I know how to do the log file, but it doesn't give me any progress information, which is why I am asking.

    Perhaps getting progress information while it runs in the background from a log file is the wrong format, but I am trying to get myself in the right direction.

    But thank you for your 'useful' comment.

  5. #5
    Join Date
    Feb 2008
    Location
    Chicago, IL
    Beans
    112
    Distro
    Ubuntu 12.10 Quantal Quetzal

    Re: rsync and progress in log files

    Quote Originally Posted by papibe View Post
    A couple of ideas:
    • use rsync with the verbose option, and redirect its output to a log that you can check.
    • get your script inside a deattached virtual terminal (using the program 'screen'), so you can attached it any time you need to see what is going on.

    Hope it helps, and tell us if you need more details on those ideas.
    Regards.
    Hmm.....any tips on the deattached virtual terminal? Will that allow me to see the progress of the already executed cron job?

  6. #6
    Join Date
    Sep 2006
    Beans
    7,193
    Distro
    Lubuntu Development Release

    Re: rsync and progress in log files

    Quote Originally Posted by diskmaster23 View Post
    I know how to do the log file, but it doesn't give me any progress information, which is why I am asking.
    You're not really going to be able to capture meaningful output from --progress. What is missing that -v does not give you?

  7. #7
    Join Date
    Feb 2008
    Location
    Chicago, IL
    Beans
    112
    Distro
    Ubuntu 12.10 Quantal Quetzal

    Re: rsync and progress in log files

    Quote Originally Posted by Lars Noodén View Post
    You're not really going to be able to capture meaningful output from --progress. What is missing that -v does not give you?
    The -v writes in the log when when a file is completed transferring, but I have no idea where the file is while it is transferring. I probably won't be watching the cron job all the time, but I would like to peek into the process and find out where it is in transferring the data; I am still not sure the log file is the right tool, probably isn't.

  8. #8
    Join Date
    Feb 2012
    Beans
    18

    Re: rsync and progress in log files

    Quote Originally Posted by diskmaster23 View Post

    But thank you for your 'useful' comment.
    Not a problem. Im glad it helped, as I said it would.

    pabibe is on the money with screen

    Or, check out these options: http://www.debianadmin.com/rsync-bac...gui-tools.html

    Their monitoring capabilities are probably not as in depth as you require.

    In my experience, progress has only been useful when I was trying to understand rsync.
    When rsync crashes, the best clue has usually been what file it stopped on (or whether it actually got started on a file at all).
    -v will log the last file it was attempting. Usually the file it crashes on is a large file.

    Best way to have rsync 'catchup' on it's self, given multiple failures, is to use --inplace, or --partial. This way, it eventually gets there.

    Look at that, my own lack of effort has caused my own self to put in more effort.

  9. #9
    Join Date
    Jan 2012
    Beans
    342

    Re: rsync and progress in log files

    diskmaster23 ..

    its fairly simple ...

    Code:
    rsync -args --logfile=/home/logs/rsync.log source dest
    The cronjob would need to be run under the user who can write to --logfile of course.

    You could then view the file as and when you wanted to see what was happening

    Code:
    ssh user@host "tail -f  ~/logs/rsync.log"
    HTH ...

  10. #10
    Join Date
    Nov 2008
    Location
    Metro Boston
    Beans
    8,519
    Distro
    Kubuntu 14.04 Trusty Tahr

    Re: rsync and progress in log files

    It sounds like you're looking for a "progress meter" or the like which would say you've moved 53% of the files or give the number of bytes transferred and remaining. Rsync doesn't support this function as far as I know.

    As others have said, if you use "-v" you'll get a list of files as they are sent that you can track with "tail -f /var/log/rsync" or wherever you direct the output. The "--stats" parameter provides summary statistics but not a progress measure.

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •