sync files (very huge directory of files)

[geoffm]

I don't know about only a work around because that actually seems pretty simple and useful. I never realized you could copy only updated files (-u), so cool. I think I'll give it a go.

thanks geoffm.

Careful, though. I just realized that when it "updates" a file, it sets the destination file's date as the current date instead of source file date. So if you're syncing more than 3 folders it might f*** you over.

[dtfinch]

rsync has a --backup option to preserve old revisions. By default it'll rename them. You can use --backup-dir to move old revisions to a different folder instead, and --delete to move deleted files the same as old revisions. You also want to add --times so that rsync will copy modification dates and use them to determine what needs to be updated, otherwise it compares every file. And if the source and destination are different filesystems (like fat32 on the usb), you'll want to use --modify-window to specify how many seconds tolerance to use when comparing times, because different filesystems round them differently.

We use rsync to back up our all file servers, millions of files, keeping past revisions. Though I'm not at work so I don't have the exact options we use.

http://www.samba.org/ftp/rsync/rsync.html

[ClarkePeters]

dtfinch,
thanks so much for the rsync suggestion.

I tried out rsync with grsync gui and it worked perfectly. What amazed me was the speed at which it operated even on the first initial sync. It went through the 73,000 files (36,500 in each directory) in only 40 seconds (that was according to rsync's computer timer, but it sure felt like only 25 seconds). On the second run, after the initial sync, it took only 6 seconds. (for anyone reviewing this, keep in mind that I was syncing to a nearly identical folder as the source not to an empty one).

This was cool because I could run the simulation first to see how it would work before actually executing. (it was also a way to "try out" the program before committing)

Also, thanks for the other tips because I think I'll be using the --backup option for my programming code, which I've yet to sync. Don't know if I can do that in the gui but I can drop back to the terminal easily. (there's a "make backups" in the advance options in the gui but I haven't checked if that's the same as the --backup option from the command line, though I suspect it is).

Thanks again and thanks a heep!

edited: changed 100,000 to 73,000 (I had forgotten that I did a major clean up and also used fslint to eliminate hords of duplicate files before I tested rsync - grsync)

[ClarkePeters]

Oh, and John Bean, I forgot to thank you too for getting me in that direction (rsync/grsync). When dtfinch said they were using it for millions of files, that's what gave me the extra shove to give it a try.
Thanks to you both.