rsync and progress in log files

[diskmaster23]

I have setup a rsync cron job to sync some data from my server to my local box, but it is alot of data. Since all this is being done via cron jobs, the process runs in the background, but I would like to be able to track its progress.

How do I track the progress of this cron job while rsync syncs in the background?

EDIT
I have already implemented -v option and -P option, but it only shows when the file was completed in the log file. The -P option only shows up when you run it at the terminal, not as a cron job. I need a way to peering into the process itself if the -v -P isn't giving me up to the second information of the remaining bytes of the file(s) being transferred in the log file.

[sleepyhollows]

What a wonderfully liberal forum.

Certainly no Stack Exchange (from where I was banned, apparently just for wanting to get involved and help...)

Just search on the web for how to schedule rsync jobs on linux. You will likely want email notification as well.

Any solution will do, it's not specific to ubuntu.

And before getting narky that I did not actually answer, take a breath and appreciate my answer for it will likely have the magic effect of causing another to come along and go one step further and actually provide you the direct answer

[papibe]

A couple of ideas:

• use rsync with the verbose option, and redirect its output to a log that you can check.
• get your script inside a deattached virtual terminal (using the program 'screen'), so you can attached it any time you need to see what is going on.

Hope it helps, and tell us if you need more details on those ideas.
Regards.

[diskmaster23]

What a wonderfully liberal forum.

Certainly no Stack Exchange (from where I was banned, apparently just for wanting to get involved and help...)

Just search on the web for how to schedule rsync jobs on linux. You will likely want email notification as well.

Any solution will do, it's not specific to ubuntu.

And before getting narky that I did not actually answer, take a breath and appreciate my answer for it will likely have the magic effect of causing another to come along and go one step further and actually provide you the direct answer

I know how to do the log file, but it doesn't give me any progress information, which is why I am asking.

Perhaps getting progress information while it runs in the background from a log file is the wrong format, but I am trying to get myself in the right direction.

But thank you for your 'useful' comment.

[diskmaster23]

A couple of ideas:

• use rsync with the verbose option, and redirect its output to a log that you can check.
• get your script inside a deattached virtual terminal (using the program 'screen'), so you can attached it any time you need to see what is going on.

Hope it helps, and tell us if you need more details on those ideas.
Regards.

Hmm.....any tips on the deattached virtual terminal? Will that allow me to see the progress of the already executed cron job?

[Lars Noodén]

I know how to do the log file, but it doesn't give me any progress information, which is why I am asking.

You're not really going to be able to capture meaningful output from --progress. What is missing that -v does not give you?

[diskmaster23]

You're not really going to be able to capture meaningful output from --progress. What is missing that -v does not give you?

The -v writes in the log when when a file is completed transferring, but I have no idea where the file is while it is transferring. I probably won't be watching the cron job all the time, but I would like to peek into the process and find out where it is in transferring the data; I am still not sure the log file is the right tool, probably isn't.

[sleepyhollows]

But thank you for your 'useful' comment.

Not a problem. Im glad it helped, as I said it would.

pabibe is on the money with screen

Or, check out these options: http://www.debianadmin.com/rsync-bac...gui-tools.html

Their monitoring capabilities are probably not as in depth as you require.

In my experience, progress has only been useful when I was trying to understand rsync.
When rsync crashes, the best clue has usually been what file it stopped on (or whether it actually got started on a file at all).
-v will log the last file it was attempting. Usually the file it crashes on is a large file.

Best way to have rsync 'catchup' on it's self, given multiple failures, is to use --inplace, or --partial. This way, it eventually gets there.

Look at that, my own lack of effort has caused my own self to put in more effort.

[Khayyam]

diskmaster23 ..

its fairly simple ...

rsync -args --logfile=/home/logs/rsync.log source dest

The cronjob would need to be run under the user who can write to --logfile of course.

You could then view the file as and when you wanted to see what was happening

ssh user@host "tail -f  ~/logs/rsync.log"

HTH ...

[SeijiSensei]

It sounds like you're looking for a "progress meter" or the like which would say you've moved 53% of the files or give the number of bytes transferred and remaining. Rsync doesn't support this function as far as I know.

As others have said, if you use "-v" you'll get a list of files as they are sent that you can track with "tail -f /var/log/rsync" or wherever you direct the output. The "--stats" parameter provides summary statistics but not a progress measure.