Page 1 of 3 123 LastLast
Results 1 to 10 of 28

Thread: scp only recent files... script help

  1. #1
    Join Date
    Jun 2010
    Location
    Portugal
    Beans
    337
    Distro
    Ubuntu 10.04 Lucid Lynx

    scp only recent files... script help

    Hi there,

    I'm trying to create a bash script to copy only recent files from a remote machine... I usually loging into the machine throw ssh and copy the file name that I want to download to my computer, and than I use scp to copy it to my machine....
    I would like to change this procedure by creating a bash script that only download the files meeting a certain criteria (e.g name, date, size)... is there any way of making this?

    Thanks

  2. #2
    Join Date
    Sep 2006
    Beans
    8,627
    Distro
    Ubuntu 14.04 Trusty Tahr

    find

    Well, find can show you files matching a certain name, date, or size.

    If you a key for authentication, then Ubuntu has a key agent that you can use to avoid typing the passphrase more than once. Here is an approximation, it will need tweaking to work:

    Code:
    # find files larger than 100KB and copy them
    find /some/path -size +100k -exec scp -i /path/to/key "{}" user@remotehost.org:. \;

  3. #3
    Join Date
    Jan 2012
    Beans
    342

    Re: scp only recent files... script help

    JCM_Pico ...

    I would say scp is the wrong tool, rsync is really designed with this kind of task in mind. It can use ssh as the transport ('rsync -e ssh'), sync only changed files, do incremental backups, exclusions, etc, etc.

    It could provide examples, but a simple google will give you some idea of how to set about using it.

    HTH ...

  4. #4
    Join Date
    Jun 2010
    Location
    Portugal
    Beans
    337
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: scp only recent files... script help

    Quote Originally Posted by Khayyam View Post
    JCM_Pico ...

    I would say scp is the wrong tool, rsync is really designed with this kind of task in mind. It can use ssh as the transport ('rsync -e ssh'), sync only changed files, do incremental backups, exclusions, etc, etc.

    It could provide examples, but a simple google will give you some idea of how to set about using it.

    HTH ...
    I know about rsync... but I have many files in the remote computer.... and I dont have them I my computer.... I just want to get from the remote server a file that have some atributes... like the date... every one started by A... things like that.... So rsync is not the solution...

    I'll give a try to the suggestion of Lars Noodén... its seam that is the right way....

  5. #5
    Join Date
    Jan 2012
    Beans
    342

    Re: scp only recent files... script help

    JM_Pico ..

    Somehow I read "from a" as "to a" .. that'll taught me to read propa before jumping in with advice .. heh.

    Still, I think rsync can be used for your purpose. A script running on the remote machine, or run via 'ssh -c', executes 'rsync --dry-run' to check for attributes your looking for (recent changes, ownership, and what-have-you) and dumps it to a file or inotify sends you its output. This can then be tagged, parsed, etc, and used as input to scp.

    As 'rsync' has the ability to log and keep stats (--log-file --log-file-format --stats) it has the advantage over 'find' in that it can be configured to use these logs to "remember" prior sync's which might be useful for regular usage.

    Anyhow, it depends on how much effort you want to put into writting your script, the example from Lars is probably exactly what your looking for, so enough from me ..

    best ...

  6. #6
    Join Date
    Jun 2010
    Location
    Portugal
    Beans
    337
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: scp only recent files... script help

    Quote Originally Posted by Khayyam View Post
    JM_Pico ..

    Somehow I read "from a" as "to a" .. that'll taught me to read propa before jumping in with advice .. heh.

    Still, I think rsync can be used for your purpose. A script running on the remote machine, or run via 'ssh -c', executes 'rsync --dry-run' to check for attributes your looking for (recent changes, ownership, and what-have-you) and dumps it to a file or inotify sends you its output. This can then be tagged, parsed, etc, and used as input to scp.

    As 'rsync' has the ability to log and keep stats (--log-file --log-file-format --stats) it has the advantage over 'find' in that it can be configured to use these logs to "remember" prior sync's which might be useful for regular usage.

    Anyhow, it depends on how much effort you want to put into writting your script, the example from Lars is probably exactly what your looking for, so enough from me ..

    best ...

    I liked the way you think ... This way I'll learn more.....
    I'm not familiarized with inotify can you explain me some more about the procedure....

    What I understand is to create a a dry run with rsync - this will tell me the files that are in the remote computer folder.... then I can use "something?" to identify the file that I want and then use scp to copy it... right?

  7. #7
    Join Date
    Jun 2008
    Location
    Ballard
    Beans
    2,395
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: scp only recent files... script help

    You can use rsync in either direction. The basic format would be thus:

    Code:
    rsync -arguments --more-arguments user@remoteA:/path/to/source/folder/or/file(s) user@remoteB:/path/to/source/folder/or/file(s)
    Depending on permissions sudo may be used/required. (If they username is the same across systems user@ may be omitted.)

    Oh, and using rsync is much better than using scp because rsync checks its work.
    "We're all in this together, kid." --H. Tuttle (a.k.a. H. Buttle)
    "Maybe it's a layer 8 problem." --thatguruguy
    A High-Tech Blech!

  8. #8
    Join Date
    Jan 2012
    Beans
    342

    Re: scp only recent files... script help

    Quote Originally Posted by JCM_Pico View Post
    [...]I'm not familiarized with inotify can you explain me some more about the procedure...
    Well, inotify is an (in kernel) "filesystem notification technology" that "provides [the] possibility [to] monitor various events on files in filesystems". Applications (like rsync) can make use of it for gauging when they should perform back-ups, etc.

    The link I provided previously was a mistake I'd ment to link to rsync-inotify but had inserted the link to the manpage instead (the links provided in this post should give you a better idea of what inotify can do). I was also a little vague stating how it might tie into your "script", the examples on the the inotify-tools wiki provide some good examples. What I'd intended to say was, your script (or some combination of scripts, rsync, and "glue") could monitor (via inotify) the filesystem, run, provide a report on changes and (ultimately) provide you with a "list" of files to pass to 'scp'. Most of this seems doable, but as I said it'll take some work to pull it together.

    Quote Originally Posted by JCM_Pico View Post
    What I understand is to create a a dry run with rsync - this will tell me the files that are in the remote computer folder.... then I can use "something?" to identify the file that I want and then use scp to copy it... right?
    rsync-inotify (or a tool similar to those provided on the inotify-tools wiki) monitors for a change in filesystem, it then carrys out a task, 'rsync --dry-run for example, and a creates a report of file changes, this is then dumped in a file and/or emailed to you. A seperate script on you client machine parses this file and either passes the list to scp or prompts you for input "Download: /files/newfile.txt [yes|no]?".

    As this is scripted there are any number of additional steps that could be included, its just a matter of gluing them together.

    The only assumption I'm making is that the remote host runs Linux and is running a kernel with inotify enabled.

    HTH ..

  9. #9
    Join Date
    Sep 2006
    Beans
    8,627
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: scp only recent files... script help

    Quote Originally Posted by jamesisin View Post
    Code:
    rsync -arguments --more-arguments user@remoteA:/path/to/source/folder/or/file(s) user@remoteB:/path/to/source/folder/or/file(s)
    It'd be really cool if both could be remote but the source and destination cannot both be remote. Just one or the other or both local. A work-around might be to go via SSHFS for one or both of them.

  10. #10
    Join Date
    Jun 2008
    Location
    Ballard
    Beans
    2,395
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: scp only recent files... script help

    It's true: when using rsync by itself either the source or the destination must be local (and either can be remote), but they cannot both be remote. If you want to do a remote-to-remote sync then you can ssh into one of those systems. (I showed both as remote just to display how each syntax would appear.)
    "We're all in this together, kid." --H. Tuttle (a.k.a. H. Buttle)
    "Maybe it's a layer 8 problem." --thatguruguy
    A High-Tech Blech!

Page 1 of 3 123 LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •