Page 2 of 2 FirstFirst 12
Results 11 to 14 of 14

Thread: sync files (very huge directory of files)

  1. #11
    Join Date
    Oct 2008
    Location
    Quebec
    Beans
    105
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: sync files (very huge directory of files)

    Quote Originally Posted by ClarkePeters View Post
    I don't know about only a work around because that actually seems pretty simple and useful. I never realized you could copy only updated files (-u), so cool. I think I'll give it a go.

    thanks geoffm.
    Careful, though. I just realized that when it "updates" a file, it sets the destination file's date as the current date instead of source file date. So if you're syncing more than 3 folders it might f*** you over.
    DELL XPS M1330 Core2
    Ubuntu 12.04 64

  2. #12
    Join Date
    Feb 2005
    Location
    Oregon
    Beans
    496
    Distro
    Ubuntu 10.04 Lucid Lynx

    Re: sync files (very huge directory of files)

    rsync has a --backup option to preserve old revisions. By default it'll rename them. You can use --backup-dir to move old revisions to a different folder instead, and --delete to move deleted files the same as old revisions. You also want to add --times so that rsync will copy modification dates and use them to determine what needs to be updated, otherwise it compares every file. And if the source and destination are different filesystems (like fat32 on the usb), you'll want to use --modify-window to specify how many seconds tolerance to use when comparing times, because different filesystems round them differently.

    We use rsync to back up our all file servers, millions of files, keeping past revisions. Though I'm not at work so I don't have the exact options we use.

    http://www.samba.org/ftp/rsync/rsync.html

  3. #13
    Join Date
    Sep 2006
    Location
    U.S. citizen in Taiwan
    Beans
    114
    Distro
    Ubuntu

    Re: sync files (very huge directory of files)

    dtfinch,
    thanks so much for the rsync suggestion.

    I tried out rsync with grsync gui and it worked perfectly. What amazed me was the speed at which it operated even on the first initial sync. It went through the 73,000 files (36,500 in each directory) in only 40 seconds (that was according to rsync's computer timer, but it sure felt like only 25 seconds). On the second run, after the initial sync, it took only 6 seconds. (for anyone reviewing this, keep in mind that I was syncing to a nearly identical folder as the source not to an empty one).

    This was cool because I could run the simulation first to see how it would work before actually executing. (it was also a way to "try out" the program before committing)

    Also, thanks for the other tips because I think I'll be using the --backup option for my programming code, which I've yet to sync. Don't know if I can do that in the gui but I can drop back to the terminal easily. (there's a "make backups" in the advance options in the gui but I haven't checked if that's the same as the --backup option from the command line, though I suspect it is).

    Thanks again and thanks a heep!

    edited: changed 100,000 to 73,000 (I had forgotten that I did a major clean up and also used fslint to eliminate hords of duplicate files before I tested rsync - grsync)
    Last edited by ClarkePeters; May 11th, 2010 at 07:46 PM. Reason: clear up misinformation
    the power of X: xhtml, xforms, xslt, xml

  4. #14
    Join Date
    Sep 2006
    Location
    U.S. citizen in Taiwan
    Beans
    114
    Distro
    Ubuntu

    Re: sync files (very huge directory of files)

    Oh, and John Bean, I forgot to thank you too for getting me in that direction (rsync/grsync). When dtfinch said they were using it for millions of files, that's what gave me the extra shove to give it a try.
    Thanks to you both.
    the power of X: xhtml, xforms, xslt, xml

Page 2 of 2 FirstFirst 12

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •