Results 1 to 10 of 10

Thread: Make Rsync synchronize file just one time

  1. #1
    Join Date
    Feb 2013
    Beans
    3

    Make Rsync synchronize file just one time

    Hi everybody.

    So here is my issue.

    I have a script to make Rsync synchronise some of my file between two remote servers in a cron job

    But when i cut a file from the destination one, rsync will re synchronize it at the next time the script will be launch by cron.
    Is there is a way (directly in rsync or in my shell script) to tell it, if you already threat this file, don't synchronize it again, even if it's not on the destination server anymore ?


    thx for your future answered

  2. #2
    Join Date
    Feb 2009
    Location
    Dallas, TX
    Beans
    7,790
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: Make Rsync synchronize file just one time

    Hi zabimaru27. Welcome to the forums

    I'm afraid after reading your post several time, I can't quite understand your concern.

    Could you post your script?

    Could you explain it with an example?

    Regards.

    P.S.:
    Quote Originally Posted by zabimaru27 View Post
    But when i cut a file from the destination ...
    You mean cut as in deleted, or that you edited it and removed part of it?

    Quote Originally Posted by zabimaru27 View Post
    ... destination one,...
    How many destinations do you have?

  3. #3
    Join Date
    Feb 2013
    Beans
    3

    Re: Make Rsync synchronize file just one time

    Hi, and thx for your answer.

    My script is quite simple

    Code:
    if [ -e /volume1/application/automate/rsyncjob.lock ]
    then
      exit
    fi
    
    touch /volume1/application/automate/rsyncjob.lock
    
    rsync -auv ******@*****.*******.com:/media/sde1/home/blackmessor/priv /volume1/telecharger
    
    #delete lock file at end of your job
    
    rm -f /volume1/application/automate/rsyncjob.lock

    So , my concern is i have two servers.
    Server SOURCE and server DESTINATION

    My script works well, and it synchronises SOURCE to DESTINATION

    but when i get my files in my DESTINATION server (which is a local server in my home), i would like to move this files to put them in my own folders to organize them.

    But if i do that, the next time the sript will be launch, rsync will detect the file is not on DESTINATION anymore (as i moved it) and will resynchronize it again.

    I want to avoid that, like if it could remember (I already synchronise this file once, so i don't do it again, even if it disapear from DESTINATION)

    I think it's not possible just with rsync, but maybe it is possible to my script to add each file synchronise with rsync in the exclusion file of rsync ???? Just an idee


    Thx

  4. #4
    Join Date
    Nov 2011
    Location
    /dev/root
    Beans
    Hidden!

    Re: Make Rsync synchronize file just one time

    If you want to 'move these files to put them in your own folders to organize them', and those folders are in the same partition, you can use hard links. That means that the files reside on the same (one and only place) in the file system, but two (or more) addresses point at them.

    This way you can keep the files where rsync put them, and at the same time organize them in folders as you like, and rsync should see that they are still there.

    Try this method, I think it will work

    See
    Code:
    info ln
    to learn about hard links (as opposed to symbolic links).

  5. #5
    Join Date
    May 2008
    Location
    United Kingdom
    Beans
    5,261
    Distro
    Ubuntu

    Re: Make Rsync synchronize file just one time

    Quote Originally Posted by sudodus View Post
    … you can use hard links.
    That is a good idea, but only on two conditions.

    1. The destination and your home folder are on the same partition. Otherwise, you can use a symbolic link, which would work as well.
    2. You do not change the file once you have copied it to your home folder (on either the source or the destination). Otherwise, rsync will synchronise it again.

    It would be possible to write a script to remember synchronised files, but this is quite a complex solution.

    Alternatively, you can use the --exclude option for rsync, but you'd have to keep it manually updated. To simplify things, you can use --exclude--from instead of --exclude.
    Always make regular backups of your data (and test them).
    Visit Full Circle Magazine for beginners and seasoned Linux enthusiasts.

  6. #6
    Join Date
    Feb 2013
    Beans
    3

    Re: Make Rsync synchronize file just one time

    thx for your answers

    Alternatively, you can use the --exclude option for rsync, but you'd have to keep it manually updated. To simplify things, you can use --exclude--from instead of --exclude.
    This is the solution i thought, but i don't know how to get the output of the file name when rsync synchronise it.

  7. #7
    Join Date
    Nov 2011
    Location
    /dev/root
    Beans
    Hidden!

    Re: Make Rsync synchronize file just one time

    Edit: You can perform a 'dry verbose run' with
    Code:
    rsync -avn
    right after you have moved your files. That will show which files it 'would like to sync', and you can add them to your exclude list.

    But I still think it is a better idea to link your files instead of moving them
    Last edited by sudodus; February 22nd, 2013 at 07:40 AM. Reason: I've been thinking about alternatives

  8. #8
    Join Date
    May 2008
    Location
    United Kingdom
    Beans
    5,261
    Distro
    Ubuntu

    Re: Make Rsync synchronize file just one time

    Quote Originally Posted by zabimaru27 View Post
    … but i don't know how to get the output of the file name when rsync synchronise it.
    I thought you wanted to exclude a file only after you have moved it away from the destination? If that is the case, you'll know the file name, because you moved it. Manually add it to the file that you are using with --exclude-from. Remember to use the full path, otherwise you may accidentally exclude like-named files from other paths.
    Always make regular backups of your data (and test them).
    Visit Full Circle Magazine for beginners and seasoned Linux enthusiasts.

  9. #9
    Join Date
    Jun 2006
    Location
    Brisbane Australia
    Beans
    713

    Re: Make Rsync synchronize file just one time

    OP, regarding your script, I would put that lock file in /tmp so that it is ensured it is deleted after a system boot. Also look at the bash builtin "trap" command to ensure the lock file gets deleted for all script exit conditions. And add --delete to your rsync command if you really want to synchronise the source to destination.

  10. #10
    Join Date
    May 2008
    Location
    United Kingdom
    Beans
    5,261
    Distro
    Ubuntu

    Re: Make Rsync synchronize file just one time

    Quote Originally Posted by markbl View Post
    OP, regarding your script, I would put that lock file in /tmp so that it is ensured it is deleted after a system boot. Also look at the bash builtin "trap" command to ensure the lock file gets deleted for all script exit conditions. And add --delete to your rsync command if you really want to synchronise the source to destination.
    All excellent suggestions, especially the trap command.

    I would add that to reduce the time to create the temporary file (and reduce the chance of a race condition failing), place it in RAM instead of /tmp. The RAM folder /run/shm already exists and is available for use.

    An even better option for locking: use flock, which is designed explicitly for this purpose.
    Always make regular backups of your data (and test them).
    Visit Full Circle Magazine for beginners and seasoned Linux enthusiasts.

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •