Page 1 of 9 123 ... LastLast
Results 1 to 10 of 85

Thread: HOWTO: Share updates across multiple machines

  1. #1
    Join Date
    Oct 2006
    Location
    Second star to the right
    Beans
    607
    Distro
    Ubuntu 7.10 Gutsy Gibbon

    HOWTO: Share updates across multiple machines

    NOTE:
    Apt-cacher does not seem to work with Lucid. This guide now use apt-cacher-ng. If you followed this guide before and are having troubles on Lucid, then run the following two commands:
    Code:
    sudo apt-get remove apt-cacher
    sudo apt-get install apt-cacher-ng
    If you have any troubles, check that configuration files are present and correct. They should remain intact, but you never know.



    So, you have Ubuntu, you love Ubuntu, and you've spread the love throughout your home. Perhaps you have a few family members using it happily, or maybe you simply enjoy having a different computer in every room of the house.

    Whatever the reason, you have multiple machines running this fine operating system. One problem, though: all those updates! You love getting upgraded software, but having to download it so many times really eats away at your bandwidth.

    Luckily for you, you're using open source software. You're not alone in this concern, and a solution already exists.

    This guide will get you setup with apt-cacher, a tool that caches Debian package files for your home network. When one of your machines requests a package, apt-cacher will check to see if it has it, and if it does, send it back to the client without downloading a single byte. If not, then it goes to the repository and grabs the package, streaming it back to the client as it does, and keeps it for future use.

    Things to note:
    1. A repository needn't be specified on the server machine for a client to access it. The server acts as a proxy which keeps a copy for itself.
    2. This is not Ubuntu-specific. Any Debian-based distro should be able to act as both a server and a client, although you may have to adapt some of the steps for yourself.


    Part Zero: The Formalities
    This has been confirmed as working on very diverse networks, with machines running different distros (not merely Ubuntu derivative) with very different architectures. I've tested this myself on versions up to, but not including, Lucid. The use of apt-cacher-ng yields similar results according to other users.

    Information for this guide was gathered primarily from the apt-cacher man page. I consulted this article (http://www.linuxjournal.com/article/1058) for help with inetd. It's rather dated, but quite useful.

    Thanks to the following users:
    omegamormegil, for providing a much easier method of redirecting APT.
    bugmenot_user, for figuring out how to clean the regular APT cache.
    mssever, for providing the workaround for upgrades.
    Skip Da Shu, for being generally awesome and providing a lot of testing.


    Part One: The Server
    You'll need to select a server for this setup. It should have a fair bit of space to spare, particularly if the client machines belong to an eclectic group. For those with more advanced partitioning schemes, the cache is kept in /var/cache/apt-cacher by default, but this can be changed.

    To get the ball rolling, we install the software in question:
    Code:
    sudo apt-get install apt-cacher-ng
    With it successfully installed, we have to tell it how to start. There are two options for this. You can have it run in daemon mode (always on), or you can get inetd, which is a program to launch server applications on request. The obvious advantage is that you save memory and CPU power, but CPU usage will spike on startup when using inetd, and you may experience a delay.

    If you're expecting a lot of traffic, go with the daemon mode. If you're a home user, chances are you'll want to go with inetd.

    Daemon Mode
    This is really quite easy. First, we'll need to edit our services file:
    Code:
    sudo nano /etc/services
    The following lines can go anywhere really, but the end of the file is convenient, particularly if you're planning to have many services running from this machine:
    Code:
    apt-cacher           3142/tcp
    apt-cacher           3142/udp
    3142 is the default port. I see no reason to change it, but if you have one, you may. If so, you'll need to change the daemon_port option in /etc/apt-cacher/apt-cacher.conf

    One more file to edit:
    Code:
    sudo nano /etc/default/apt-cacher
    Set AUTOSTART to 1 to get the daemon to launch at boot time.

    To launch it now, simply type the following:
    Code:
    sudo /etc/init.d/apt-cacher start
    Now, we're all done for the server side of things. Skip down to Part Two

    Inetd Mode
    To kick things off, we'll need to install the service:
    Code:
    sudo apt-get install inetutils-inetd
    Next, we edit a couple of files:
    Code:
    sudo nano /etc/services
    Add these two lines to the end of the file:
    Code:
    apt-cacher           3142/tcp
    apt-cacher           3142/udp
    Now, we have to setup inetd:
    Code:
    sudo nano /etc/inetd.conf
    Add this line:
    Code:
    apt-cacher stream tcp nowait www-data /usr/sbin/tcpd /usr/sbin/apt-cacher -i
    Now, restart the inetd server:
    Code:
    sudo /etc/init.d/inetutils-inetd restart
    You're done!


    Part Two: The Client
    To use apt-cacher, your clients need to redirect their requests to the server machine. APT has its own proxy configuration. Thanks to omegamormegil for this tip
    Type the following in a terminal:
    Code:
    echo 'Acquire::http::Proxy "http://hostname:3142";' | sudo tee /etc/apt/apt.conf.d/01proxy
    Where hostname is a valid identifier for your server (for home users, an IP address will probably be convenient).

    For laptop users, this means that you'll be unable to update using this method while on the road (unless you have a global route to your server, although the speed will be greatly reduced). To switch back to direct updates, type the following command:
    Code:
    echo 'Acquire::http::Proxy "http://";' | sudo tee /etc/apt/apt.conf.d/01proxy
    Then once you get home, the command given above will get you using shared updates again.

    If you prefer, APT will look for the last line in the file. You can cut/paste them as needed, but the above commands can be easily written into a script to automate the process for you.

    From ace214, the following command will tell apt-get to use direct fetching methods:
    Code:
    export http_proxy=http://
    As far as I know, this will only work for apt-get until you end your session (by logging out, or exiting/closing the terminal). If you're not tied to the GUI, this may serve as a useful solution for you.

    Now, update!
    Code:
    sudo apt-get update
    If all goes well, you should download packages as usual. Try downloading some software, or getting some updates. It should work pretty much as before.

    If you are upgrading, these steps will not work. This appears to be a bug in Update Manager.

    To upgrade, you must edit the 01proxy file to use direct updates as described above. Now, run the following command:
    Code:
    sudo sed --in-place s_http://_http://hostname:3142/_ /etc/apt/sources.list
    The syntax may seem a little daunting, but this simply replaces any occurance of http:// with http://hostname:3142/. Once again, hostname must be a valid identifier for your server.

    Upgrades should now work normally through the Update Manager. Once you've finished the upgrade, run the following command:
    Code:
    sudo sed --in-place s_http://hostname:3142/_http://_ /etc/apt/sources.list
    Now edit the 01proxy file to set yourself to shared updates as described above.

    So far, this is the only workaround I know of. I apologise to anyone who may have suffered this bug. Once more, thank you mssever for bringing this to my attention and providing a fix.


    Part Three: The Transition
    Chances are your server's up to date,and has a fair package cache of its own. Why duplicate all of this fine work? Apt-cacher comes with a migration helper for pre-caching. Type the following in a terminal to copy your cache over:
    Code:
    sudo cp /var/cache/apt/archives/*.deb /var/cache/apt-cacher/import
    sudo /usr/share/apt-cacher/apt-cacher-import.pl
    At the moment, the server machine itself will go downloading its own packages, rather than reusing those in the cache. To fix this, use the same proxy setup as used above on the clients, using "localhost" as the hostname.

    The final step is optional, but it will help you free up space. The server is currently maintaining two caches: the regular apt one, and the shared cache.

    NOTE: If you omit this step, APT will keep building its own cache. This may be of benefit to you when you uninstall apt-cacher if have limited bandwidth, but APT's operation will not be affected.

    If your server uses Synaptic at all, there's an option under Settings -> Preferences -> Files to delete packages after installation. Thanks to Skip Da Shu for pointing this out to me.

    This will only apply (in Ubuntu and Xubuntu) to packages fetched through the GUI. It will not work in Kubuntu, and will not work from the command line. For a more universal solution, enter the following command:
    Code:
    echo 'DPkg::Post-Invoke {"/bin/rm -f /var/cache/apt/archives/*.deb || true";};' | sudo tee /etc/apt/apt.conf.d/clean
    This command is carried out just before APT quits, and will clean the cache of any .deb files. Thanks to bugmenot_user for this command.


    Part Four: The Removal
    To undo the changes made in this guide, you must remove the proxy configuration lines. You must also remove the file /etc/apt/apt.conf.d/clean, or set the option in Synaptic back to its original value.

    Purging inetutils-inetd, inetutils-tcpd, and apt-cacher-ng will undo most of the remaining changes. Remove the two lines from the end of /etc/services, and your system should be back to the way it was.
    Last edited by coolen; August 23rd, 2010 at 01:41 AM.

  2. #2
    Join Date
    Apr 2007
    Location
    Columbia, MD
    Beans
    69
    Distro
    Ubuntu 12.04 Precise Pangolin

    Re: HOWTO: Share updates across multiple machines

    Thanks for the info! I have apt-cacher running on my network, now. I figured out how to use a http proxy to make the clients update through the apt-cacher server, which might be easier than editing the sources.list file. It is MUCH easier, IMO, if you are on a laptop. I added the method I'm using to the end of this page:

    https://help.ubuntu.com/community/Apt-Cacher-Server

    Hope it's helpful!

  3. #3
    Join Date
    Oct 2006
    Location
    Second star to the right
    Beans
    607
    Distro
    Ubuntu 7.10 Gutsy Gibbon

    Re: HOWTO: Share updates across multiple machines

    Thanks for that. Does seem easier, especially when you've got multiple sources files.
    I don't really mind, and it's starting to get to me.

  4. #4
    Join Date
    Oct 2006
    Location
    Second star to the right
    Beans
    607
    Distro
    Ubuntu 7.10 Gutsy Gibbon

    Re: HOWTO: Share updates across multiple machines

    Seems that your method runs a fair bit faster, too.
    I don't really mind, and it's starting to get to me.

  5. #5
    Join Date
    Apr 2006
    Beans
    6

    Re: HOWTO: Share updates across multiple machines

    Wait, which was faster? The inetd proxy or the linked apache2 proxy?

  6. #6
    Join Date
    May 2005
    Location
    Yoshi Island
    Beans
    514
    Distro
    Ubuntu 9.10 Karmic Koala

    Re: HOWTO: Share updates across multiple machines

    Ok sorry now if I put apt-cacher on my 'server' and then use a proxy for all my desktop clients, does this mean only cached packages are downloaded on the clients??

  7. #7
    Join Date
    Oct 2006
    Location
    Second star to the right
    Beans
    607
    Distro
    Ubuntu 7.10 Gutsy Gibbon

    Re: HOWTO: Share updates across multiple machines

    To bryceman, I haven't tried the apache2 method, I'm afraid. I don't currently have any need for a web server, and it seems more straightforward to me to set this up on its own. I'll add it in the future for those interested, or if anyone's interested in contributing, I can get it up sooner.
    When I mentioned that in the post, I was referring to the old proxy method, in which you had to edit each line of your sources list. My download speeds were generally about 10-20% of the speed of a direct download, and are usually running pretty close to normal speeds using APT's proxy configuration.

    To spockrock, this does not in any way limit your experience. You can still download any package you want, from any repository you want. Once you have this set up, you shouldn't need to think about it unless you're on a laptop moving between networks.
    What happens is that the request goes to the server machine, where the apt-cacher service will pick it up. It'll download the package, streaming a copy of it to you (the client), and keeping one for itself. If another client requests the same package, the service will simply send them a copy of the cached package.
    I don't really mind, and it's starting to get to me.

  8. #8
    Join Date
    May 2005
    Location
    Yoshi Island
    Beans
    514
    Distro
    Ubuntu 9.10 Karmic Koala

    Re: HOWTO: Share updates across multiple machines

    ok cool so its fine then the servers sources.lst and clients sources.lst files are different, as well if the client upgrades or installs a package the server machine will then use the cached package as well right?
    Last edited by spockrock; November 7th, 2007 at 02:12 PM.

  9. #9
    Join Date
    Oct 2006
    Location
    Second star to the right
    Beans
    607
    Distro
    Ubuntu 7.10 Gutsy Gibbon

    Re: HOWTO: Share updates across multiple machines

    Yep. The next time the server requests the new package (uhh, from itself), it'll send itself the cached one without touching the 'net.
    I don't really mind, and it's starting to get to me.

  10. #10
    Join Date
    Nov 2006
    Location
    Pretoria, SA
    Beans
    98
    Distro
    Ubuntu 9.04 Jaunty Jackalope

    Re: HOWTO: Share updates across multiple machines

    All sounds good, but will this work if the server is running 6.06 and the desktops are running anything later?

Page 1 of 9 123 ... LastLast

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •