Page 1 of 2 12 LastLast
Results 1 to 10 of 13

Thread: copy only files that have changed?

  1. #1
    Join Date
    Apr 2007
    Location
    South Carolina
    Beans
    850
    Distro
    Ubuntu Studio 14.04 Trusty Tahr

    copy only files that have changed?

    Is there a way to copy only files that have changed since the last backup?
    I know the -u switch will update existing files but that is not what I want to do. I'm working on a web site & I have about 20 or 30 different files that I constantly back up into a new folder that is named by the date & time. I use a shell script that makes it easy to do but I can't figure out how to only backup the files I've worked on since the last backup. I may only work on one or two of the files at a time but since they get backed up into an empty directory the -u switch still copies all the files. In DOS xcopy there is a switch that will do this but I can't find anything in Linux that will. I find it hard to believe that MSDOS has got something better from the command line than Linux. Surely I must be missing something, right?
    Gary

    A wise man knows what he doesn't know.

  2. #2
    Join Date
    Feb 2009
    Location
    Dallas, TX
    Beans
    7,647
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: copy only files that have changed?

    That is basically what rsync do best. For example, the first time you run this:
    Code:
    rsync -av source/ destination/
    It will copy everything (mirror) from source to destination. But the 2nd time:
    Code:
    rsync -av source/ destination/
    It will only copy the files that have changed.

    Does that help? There are a lot of more options. Tell us if you need more help customizing it.

    Regards.

    EDIT: some of the options include a way to access a backup that you already have done, so you can compare the source with the backup and only copy the changed files into a different directory.
    Last edited by papibe; December 14th, 2011 at 08:27 AM.

  3. #3
    Join Date
    Apr 2007
    Location
    South Carolina
    Beans
    850
    Distro
    Ubuntu Studio 14.04 Trusty Tahr

    Re: copy only files that have changed?

    Quote Originally Posted by papibe View Post
    That is basically what rsync do best. For example, the first time you run this:
    Code:
    rsync -av source/ destination/
    It will copy everything (mirror) from source to destination. But the 2nd time:
    Code:
    rsync -av source/ destination/
    It will only copy the files that have changed.

    Does that help? There are a lot of more options. Tell us if you need more help customizing it.
    .................
    Thanks for the ideas,
    I always thought I needed to use the "-u" switch with rsync to copy only files that have changed but it does work like you said. I don't see anywhere in the man pages where it explains why it works like that with the "-a" switch but it does. Anyways that really doesn't do the job I need because each backup goes to an empty directory. So if I use
    Code:
    rsync -av
    or
    Code:
    rsync -auv
    it still copies every file each time to the new directory since there are no existing files for it to compare. What I'm trying to do is to have an archive of all my stuff so I can go back to any point in time & resurrect things the way they were in case I find some mistakes later on. Maybe if I post part of the script it will explain what I'm trying to do better:

    Code:
    #! /bin/bash
    clear
    echo""
    echo "This will back up the load html & php files to ~/load  in a directory named by time & date"
    echo ""
    read -n1 -r -p  "Press \" Y\" to continue with backup or any other key to exit" name
    clear
    x='y'
    y='Y'
    if test $x = "$name" ||test $y = "$name"
    then
    
    title=$(date +%Y-%m-%d-%T);
    
    echo ""
    mkdir ~/loadcalc/load_$title
    cp /var/www/my_web/load/* ~/load/load_$title
    ............
    Gary

    A wise man knows what he doesn't know.

  4. #4
    Join Date
    Sep 2006
    Beans
    8,627
    Distro
    Ubuntu 14.04 Trusty Tahr

    Re: copy only files that have changed?

    The -a is the same as -rlptgoD. It will copy only the files that changed.

  5. #5
    Join Date
    Apr 2007
    Location
    South Carolina
    Beans
    850
    Distro
    Ubuntu Studio 14.04 Trusty Tahr

    Re: copy only files that have changed?

    Quote Originally Posted by Lars Noodén View Post
    The -a is the same as -rlptgoD. It will copy only the files that changed.
    I understand that but what I don't understand is which of those switches tells it to copy only files that have changed. I've checked every one of those switches & I don't see any one of them that will do that. Is there something undocumented or can someone explain it to me?
    Gary

    A wise man knows what he doesn't know.

  6. #6
    Join Date
    Feb 2009
    Location
    Dallas, TX
    Beans
    7,647
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: copy only files that have changed?

    Here's something for you.

    These are a couple of scripts that I use to backup data. The first one creates a full backup. The second makes a differential backup, on a different directory. I modified them with what I think are your own directories. Replace 'your_user' with your actual username.

    full.sh:
    Code:
    #!/bin/bash
    SOURCE="/var/www/my_web/load/"
    BACKUP="/home/your_user/load/"
    
    DATE=$(date +%Y-%m-%d-%T)
    
    DESTINATION="$BACKUP"/"$DATE"-full/
    
    # the symbolic link 'latest-full' will always point to the lastest full backup.
    rm -rf "$BACKUP"/latest-full
    ln -s "$DESTINATION" "$BACKUP"/latest-full
    
    rsync -av "$SOURCE" "$DESTINATION"
    differential.sh:
    Code:
    #!/bin/bash
    SOURCE="/var/www/my_web/load/"
    BACKUP="/home/your_user/load/"
    LBACKUP="/home/your_user/load/latest-full/"
    
    DATE=$(date +%Y-%m-%d-%T)
    
    DESTINATION="$BACKUP"/"$DATE"-diff/
    
    rsync -av --compare-dest="$LBACKUP" "$SOURCE" "$DESTINATION"
    
    # delete empty directories (rsync copies the whole tree).
    cd "$DESTINATION"
    find . -depth -type d -empty -delete
    I hope they provide some ideas to you.
    Kind Regards.

  7. #7
    Join Date
    May 2007
    Beans
    121
    Distro
    Xubuntu 12.04 Precise Pangolin

    Re: copy only files that have changed?

    Another option is <gasp> pay-for software. There is an excellent program that I have used for years to keep directories in sync - it is all gui, and they wrote a linux version! I pay for a license every few years as the versions grow - it is called "Beyond Compare". The folks that wrote it did a bang-up job. Yes, you can use CLI to do the same thing - but I get tired of having to think about what I need my computer to do. Beyond Compare gives me a nice gui in either OS, I can see what I am doing, I can see what needs to be done. No muss, no fuss.

  8. #8
    Join Date
    Apr 2007
    Location
    South Carolina
    Beans
    850
    Distro
    Ubuntu Studio 14.04 Trusty Tahr

    Re: copy only files that have changed?

    Quote Originally Posted by papibe View Post
    Here's something for you.

    These are a couple of scripts that I use to backup data. The first one creates a full backup. The second makes a differential backup, on a different directory. I modified them with what I think are your own directories. Replace 'your_user' with your actual username.

    full.sh:
    Code:
    #!/bin/bash
    SOURCE="/var/www/my_web/load/"
    BACKUP="/home/your_user/load/"
    
    DATE=$(date +%Y-%m-%d-%T)
    
    DESTINATION="$BACKUP"/"$DATE"-full/
    
    # the symbolic link 'latest-full' will always point to the lastest full backup.
    rm -rf "$BACKUP"/latest-full
    ln -s "$DESTINATION" "$BACKUP"/latest-full
    
    rsync -av "$SOURCE" "$DESTINATION"
    differential.sh:
    Code:
    #!/bin/bash
    SOURCE="/var/www/my_web/load/"
    BACKUP="/home/your_user/load/"
    LBACKUP="/home/your_user/load/latest-full/"
    
    DATE=$(date +%Y-%m-%d-%T)
    
    DESTINATION="$BACKUP"/"$DATE"-diff/
    
    rsync -av --compare-dest="$LBACKUP" "$SOURCE" "$DESTINATION"
    
    # delete empty directories (rsync copies the whole tree).
    cd "$DESTINATION"
    find . -depth -type d -empty -delete
    I hope they provide some ideas to you.
    Kind Regards.
    Very interesting,
    I'm not sure what the "compare-dest=" does. The man pages are not very descriptive.
    Gary

    A wise man knows what he doesn't know.

  9. #9
    Join Date
    Apr 2007
    Location
    South Carolina
    Beans
    850
    Distro
    Ubuntu Studio 14.04 Trusty Tahr

    Re: copy only files that have changed?

    I guess if I go back to my original question what I need to know is if there is any command that can mark a file so any change to the file afterwards will show a different mark. MSDOS does it by using an archive bit on the file that can be removed & any change to the file will put the archive bit back on the file. It works great for incremental backups.
    Gary

    A wise man knows what he doesn't know.

  10. #10
    Join Date
    Feb 2009
    Location
    Dallas, TX
    Beans
    7,647
    Distro
    Ubuntu 16.04 Xenial Xerus

    Re: copy only files that have changed?

    Quote Originally Posted by garyed View Post
    Very interesting,
    I'm not sure what the "compare-dest=" does. The man pages are not very descriptive.
    This is how it works:

    It all start by making a full backup (or mirror) of your source directory. If you do a regular rsync again, you would update the destination, but the previous version of the backup-ed data would be overwritten.

    Another approach is to do a differential backup: leave the full backup intact, and create a new directory with only the content that has change on the source.

    A typical rsync implementation has this structure:
    Code:
    rsync -av --compare-dest=full_back_up  original_source/   new_differential_backup/
    rsync will follow this logic: (1) go to original_source and take the first file; (2) check if the file exist in the full_backup; (3) if exist and haven't change, stop and fetch the next file from original_source; (4) if it does not exist or has changed, then copy it into new_differential_backup.

    Does that clarify things a little bit?
    Regards.

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •