Page 1 of 2 12 LastLast
Results 1 to 10 of 14

Thread: Backup Amazon S3 / Scripts / Rsync?

  1. #1
    Join Date
    Apr 2005
    Location
    England
    Beans
    78
    Distro
    Ubuntu 9.04 Jaunty Jackalope

    Backup Amazon S3 / Scripts / Rsync?

    Just wondered if any of you guys are using Amazon S3 for your backups and if so, what scripts are you using? Rolling your own or is there some half decent open source solutions?

    I want to try it out for backing up flat files / MySQL dumps.

    TIA!
    IT Support Bristol
    LinuxMOZ System Admin Guides

  2. #2
    Join Date
    Mar 2007
    Location
    UK, Staffordshire
    Beans
    348
    Distro
    Ubuntu 11.04 Natty Narwhal

    Re: Backup Amazon S3 / Scripts / Rsync?

    Personally I use Jungledisk but it is not free, for FOSS I would go with duplicity. Uploading can be slow but restoring from off-site is great.
    Box 1 | Xubuntu 12.10 64 bit | Gigabyte GA-MA78GM-S3H rev2 / AMD 5900+ | 250Gb Hitachi SATA / 4Gb RAM
    Ubuntu User #14507[/U] | GPG Key 0xBE7E87FD

  3. #3
    Join Date
    Apr 2005
    Location
    England
    Beans
    78
    Distro
    Ubuntu 9.04 Jaunty Jackalope

    Re: Backup Amazon S3 / Scripts / Rsync?

    I have used JungleDisk before however it has no command line? It's gui only?
    IT Support Bristol
    LinuxMOZ System Admin Guides

  4. #4
    Join Date
    Jun 2009
    Location
    0000:0400
    Beans
    Hidden!

    Re: Backup Amazon S3 / Scripts / Rsync?

    Short Version: Whatever you choose as your backup method, stick with it. If you are going to be making substantial usage of S3, I recommend a service like s3rsync or jungledisk.

    Long Version:
    Some time in the 3Q of last year, I toyed with the idea of an rsync based backup script for S3 at my brother's request. While it worked, it was slow and mildly unreliable. I'll blame this on the connection medium (s3fs), not my script.

    rsync generates a lot of extra bandwidth determining what files it needs to transfer before it actually does the transfer, particularly when it has no rsync daemon on the other end to communicate with. In my testing, roughly the same bandwidth was used doing an update of only a few files versus blindly rsync'ing the entire bucket up to S3.

    My major concern with S3 is that a directory does not have a standard definition. I'll expand on that. While a bucket is a bucket and is defined by Amazon, a directory within a bucket does not. This is determined by your backup agent, and is not guaranteed to be (read: won't be) compatible with multiple backup agents. In other words, if you do your backups with JungleDisk and then switch to another method, you'll see an empty bucket because the new software doesn't understand the directories put in place by JD.
    Last edited by falconindy; March 19th, 2010 at 03:52 PM.

  5. #5
    Join Date
    Mar 2007
    Location
    UK, Staffordshire
    Beans
    348
    Distro
    Ubuntu 11.04 Natty Narwhal

    Re: Backup Amazon S3 / Scripts / Rsync?

    Quote Originally Posted by ksudbury View Post
    I have used JungleDisk before however it has no command line? It's gui only?
    Incorrect, the latest desktop edition has reintroduced CLI support and there is a server edition.
    Box 1 | Xubuntu 12.10 64 bit | Gigabyte GA-MA78GM-S3H rev2 / AMD 5900+ | 250Gb Hitachi SATA / 4Gb RAM
    Ubuntu User #14507[/U] | GPG Key 0xBE7E87FD

  6. #6
    Join Date
    Apr 2005
    Location
    England
    Beans
    78
    Distro
    Ubuntu 9.04 Jaunty Jackalope

    Re: Backup Amazon S3 / Scripts / Rsync?

    Quote Originally Posted by ushills View Post
    Incorrect, the latest desktop edition has reintroduced CLI support and there is a server edition.
    Ohh, I might check it out then, does it have a 5GB file size limit still?
    IT Support Bristol
    LinuxMOZ System Admin Guides

  7. #7
    Join Date
    Aug 2006
    Beans
    304
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: Backup Amazon S3 / Scripts / Rsync?

    Quote Originally Posted by ksudbury View Post
    Ohh, I might check it out then, does it have a 5GB file size limit still?
    http://www.jungledisk.com/personal/d...q/default.aspx

    What is the maximum file size that can be stored with Jungle Disk?

    There is no limit to the total number or size of files you can backup. For the Network Drive feature, there is a file size limit of 5GB. Note that package files on the Mac, such as Aperture vaults or iPhoto libraries, are actually directories made up of many files. The 5GB limit does not apply to the total package size, which can be virtually unlimited.
    This seems to have to do with the way that the files are cached on disk when using the Network Drive feature.

  8. #8
    Join Date
    Apr 2010
    Beans
    3

    Re: Backup Amazon S3 / Scripts / Rsync?

    Quote Originally Posted by ushills View Post
    Incorrect, the latest desktop edition has reintroduced CLI support and there is a server edition.
    i have been having a REAL hard time getting jungledisk setup so that i can mount it at /mnt/jungledisk and access the files as a regular (non root/non su) user. it's very hard to find instructions. could you point me somewhere or do you have any tips?

    the error message i'm getting follows:

    Failed to mount volume

    Warning Details (Jungle Disk Desktop 3.06 Linux Console i386)
    ------------------------
    xDriveMapFailed - fuse_mount(/mnt/jungledisk) failed: fuse: failed to open mountpoint for reading: Permission denied

    Exception Code: xDriveMapFailed (51)
    Time: 04/08/2010 08:28:07 PM (GMT-4)
    Detailed Message: fuse_mount(/mnt/jungledisk) failed: fuse: failed to open mountpoint for reading: Permission denied

    Error Location: JungleFuse.cpp:1468
    via JungleFuse.cpp:1569

  9. #9
    Join Date
    Apr 2010
    Beans
    3

    Re: Backup Amazon S3 / Scripts / Rsync?

    i finally got this working. i had to add myself to the fuse group, and then i had to change up the permissions on my /etc/fuse.conf file. i THINK this is what got it working ultimately.

    but it's weird...if i edit files with gedit, i cannot save them, but with nano on the command line it works fine. that's good enough for me.

  10. #10
    Join Date
    May 2005
    Location
    St. John's, NL
    Beans
    68
    Distro
    Ubuntu 13.04 Raring Ringtail

    Re: Backup Amazon S3 / Scripts / Rsync?

    I know i'm reviving an old thread here, but, I had a question that maybe someone could answer, without me starting a new thread.

    I have a home setup with a Ubuntu 10.04 server and a desktop and laptop that dual boots between windows/ubuntu.

    If I went with the jungledisk personal desktop edition, can I use it for all of my machines, the server included? Or do I need a separate one for the server (just a small server hosting a few websites, sql, etc.)?

    I'm currently using free dropbox (2GB) but i'm approaching its limit now and want to move to something a little bigger, but I don't need Dropbox's next tier of $10/month for 50GB.

    So, main question, what's the difference between the desktop edition and the server edition in my setup? Any? Will desktop edition work on all 3 systems?
    Laptop: XPS 15 L502X (Ubuntu 13.04 64bit)
    Nettop: Zotac Mag HD ND-01 (OpenELEC 3.0)
    Desktop: Windows 8 64bit / Ubuntu 13.04 64bit

Page 1 of 2 12 LastLast

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •