rsync and progress in log files

**diskmaster23** · February 12th, 2012

Originally Posted by SeijiSensei

It sounds like you're looking for a "progress meter" or the like which would say you've moved 53% of the files or give the number of bytes transferred and remaining. Rsync doesn't support this function as far as I know.

As others have said, if you use "-v" you'll get a list of files as they are sent that you can track with "tail -f /var/log/rsync" or wherever you direct the output. The "--stats" parameter provides summary statistics but not a progress measure.

It has the -P option, but it only shows while you are running the command at a terminal, not as a cron job and into the log file.

**Khayyam** · February 12th, 2012

Originally Posted by SeijiSensei

It sounds like you're looking for a "progress meter" or the like which would say you've moved 53% of the files or give the number of bytes transferred and remaining. Rsync doesn't support this function as far as I know.

Actually it does, '--progress', or the '-P' switch (I see now form the OP's edits that s/he was using '-P'). The problem is that because --progress is updating the screen there are control characters printed, so its not really designed for non-interactive shells (such as cron).

The output of --progress could be redirected to a file but then the file woud contain control characters and would need to be parsed with sed 's/\r/\n/g' to remove these chars. So, obviously thats not going to work with 'tail -f'.

So, really, what the OP wants is an interactive feature for a non-interactive shell, and that is somwhat of an ask. The closest they will get, I think, is via '--log-file='.

best ... khay

**diskmaster23** · February 12th, 2012

Originally Posted by Khayyam

Actually it does, '--progress', or the '-P' switch (I see now form the OP's edits that s/he was using '-P'). The problem is that because --progress is updating the screen there are control characters printed, so its not really designed for non-interactive shells (such as cron).

The output of --progress could be redirected to a file but then the file woud contain control characters and would need to be parsed with sed 's/\r/\n/g' to remove these chars. So, obviously thats not going to work with 'tail -f'.

So, really, what the OP wants is an interactive feature for a non-interactive shell, and that is somwhat of an ask. The closest they will get, I think, is via '--log-file='.

best ... khay

<shrugs> I figured I was asking too much, but I am not as knowledgeable as some folks around here. So I thought I would ask.

Thanks for everyone's help.

**papibe** · February 13th, 2012

Originally Posted by diskmaster23

Hmm.....any tips on the deattached virtual terminal? Will that allow me to see the progress of the already executed cron job?

It would allow you to connect to the terminal where the command it is actually being executed.

IMHO, this is very helpful when you are rsyncing a few big files, instead of several small to medium files. In this case the option --progress make more sense than -v.

I personally use a combination of screen and rsync to share family photos and videos with my brother.

Inside cron you need to create a detached screen:

Code:

screen -dmS NameOfScreen your_rsync_script.sh

At any time you can check if there are screens actives by doing:

Code:

screen -ls

If you see something like this:

Code:

There is a screen on:
	3637.NameOfScreen	(02/12/2012 08:31:02 PM)	(Detached)
1 Socket in /var/run/screen/S-user.

it means that the script is still running inside the screen named 'NameOfScreen'. In order to attach (connect) to the virtual screen do this:

Code:

screen -r NameOfScreen

To leave things running as they were, de attach from the screen by pressing Ctrl+a d (control a, and then d).

Hope it helps.
Regards.

**diskmaster23** · February 13th, 2012

Originally Posted by papibe

It would allow you to connect to the terminal where the command it is actually being executed.

IMHO, this is very helpful when you are rsyncing a few big files, instead of several small to medium files. In this case the option --progress make more sense than -v.

I personally use a combination of screen and rsync to share family photos and videos with my brother.

Inside cron you need to create a detached screen:

Code:

screen -dmS NameOfScreen your_rsync_script.sh

At any time you can check if there are screens actives by doing:

Code:

screen -ls

If you see something like this:

Code:

There is a screen on:
	3637.NameOfScreen	(02/12/2012 08:31:02 PM)	(Detached)
1 Socket in /var/run/screen/S-user.

it means that the script is still running inside the screen named 'NameOfScreen'. In order to attach (connect) to the virtual screen do this:

Code:

screen -r NameOfScreen

To leave things running as they were, de attach from the screen by pressing Ctrl+a d (control a, and then d).

Hope it helps.
Regards.

I'll give it a try.

**diskmaster23** · February 13th, 2012

Originally Posted by papibe

It would allow you to connect to the terminal where the command it is actually being executed.

IMHO, this is very helpful when you are rsyncing a few big files, instead of several small to medium files. In this case the option --progress make more sense than -v.

I personally use a combination of screen and rsync to share family photos and videos with my brother.

Inside cron you need to create a detached screen:

Code:

screen -dmS NameOfScreen your_rsync_script.sh

At any time you can check if there are screens actives by doing:

Code:

screen -ls

If you see something like this:

Code:

There is a screen on:
	3637.NameOfScreen	(02/12/2012 08:31:02 PM)	(Detached)
1 Socket in /var/run/screen/S-user.

it means that the script is still running inside the screen named 'NameOfScreen'. In order to attach (connect) to the virtual screen do this:

Code:

screen -r NameOfScreen

To leave things running as they were, de attach from the screen by pressing Ctrl+a d (control a, and then d).

Hope it helps.
Regards.

The script looks like this

Code:

#!/bin/bash
cmd="rsync -avzhP --verbose --log-file=/data/server-rsync-log --owner=david --group=admin --chmod=a+rw,g+rw,g+rw,o-wx /data/server-remote/ /data/server-local/"
eval $cmd # execute command

The problem is, when I run

Code:

screen -dmS rsync -avzhP --verbose --log-file=/data/server-rsync-log --owner=david --group=admin --chmod=a+rw,g+rw,g+rw,o-wx /data/server-remote/ /data/server-local/

I get this

Code:

Screen version 4.00.03jw4 (FAU) 2-May-06

So I thought I had to run it like this

Code:

screen -dmS /usr/bin/ssh-server-rsync

but when I do, I get this

Code:

Must run suid root for multiuser support.

What am I doing wrong?

**papibe** · February 14th, 2012

screen expects a name after the -S option, and then a command. Try this:

Code:

screen -dmS myscreen rsync -avzhP --verbose --log-file=/data/server-rsync-log --owner=david --group=admin --chmod=a+rw,g+rw,g+rw,o-wx /data/server-remote/ /data/server-local/

Although, I would rather run it like this:

Code:

screen -dmS myscreen /path/to/myrsync,sh

where myrsync.sh contains the rsync command:

Code:

#!/bin/bash
rsync -avzhP --verbose --log-file=/data/server-rsync-log --owner=david --group=admin --chmod=a+rw,g+rw,g+rw,o-wx /data/server-remote/ /data/server-local/

Let us know how it goes.
Regards.

**diskmaster23** · February 16th, 2012

Originally Posted by papibe

screen expects a name after the -S option, and then a command. Try this:

Code:

screen -dmS myscreen rsync -avzhP --verbose --log-file=/data/server-rsync-log --owner=david --group=admin --chmod=a+rw,g+rw,g+rw,o-wx /data/server-remote/ /data/server-local/

Although, I would rather run it like this:

Code:

screen -dmS myscreen /path/to/myrsync,sh

where myrsync.sh contains the rsync command:

Code:

#!/bin/bash
rsync -avzhP --verbose --log-file=/data/server-rsync-log --owner=david --group=admin --chmod=a+rw,g+rw,g+rw,o-wx /data/server-remote/ /data/server-local/

Let us know how it goes.
Regards.

I just wanted to let everyone know that it worked. Consider this thread solved.