HOWTO: Share updates across multiple machines

[coolen]

Important Note:
Instructions for upgrading have been added to Part Two. Thanks to mssever for this workaround.

Update:
Added easier, all-in-one proxy method. It's no longer necessary to alter every line of your sources.list file, and additional sources files should be handled as well. Thanks be to omegamormegil for this tip. See part two for the changes.

Instructions to automatically clean the regular APT cache are now included. Thanks to bugmenot_user for figuring this one out. Changes can be found in part three.


So, you have Ubuntu, you love Ubuntu, and you've spread the love throughout your home. Perhaps you have a few family members using it happily, or maybe you simply enjoy having a different computer in every room of the house.

Whatever the reason, you have multiple machines running this fine operating system. One problem, though: all those updates! You love getting upgraded software, but having to download it so many times really eats away at your bandwidth.

Luckily for you, you're using open source software. You're not alone in this concern, and a solution already exists.

This guide will get you setup with apt-cacher, a tool that caches Debian package files for your home network. When one of your machines requests a package, apt-cacher will check to see if it has it, and if it does, send it back to the client without downloading a single byte. If not, then it goes to the repository and grabs the package, streaming it back to the client as it does, and keeps it for future use.

Things to note:
1. A repository needn't be specified on the server machine for a client to access it. The server acts as a kind of proxy which keeps a copy for itself.
2. This is not Ubuntu-specific. Any Debian-based distro should be able to act as both a server and a client, although you may have to adapt some of the steps for yourself.

I've noticed that the new proxy setup runs significantly faster, at least for me, than the old one (close to direct download speeds). Let me know if you experience decreased performance using this howto.

Part Zero: The Formalities
Each step of this guide has been tested on my own home network between a laptop and a desktop machine, each running Feisty 32-bit. Also tested with two desktops and a laptop on Gutsy, and most recently with two desktops (Hardy and Gutsy). No errors have yet occurred due to differing versions. Skip Da Shu confirms this as working on a network with multiple derivatives, architectures and releases.

Information for this guide was gathered primarily from the apt-cacher man page. I consulted this article (http://www.linuxjournal.com/article/1058) for help with inetd. It's rather dated, but quite useful.


Part One: The Server
You'll need to select a server for this setup. It should have a fair bit of space to spare, particularly if the client machines belong to an eclectic group. For those with more advanced partitioning schemes, the cache is kept in /var/cache/apt-cacher by default, but this can be changed.

To get the ball rolling, we install the software in question:

sudo apt-get install apt-cacher

With it successfully installed, we have to tell it how to start. There are two options for this. You can have it run in daemon mode (always on), or you can get inetd, which is a program to launch server applications on request. The obvious advantage is that yo save memory and CPU power, but CPU usage will spike on startup when using inetd.
If you're expecting a lot of traffic, go with the daemon mode. If you're a home user, chances are you'll want to go with inetd.

Daemon Mode
This is really quite easy. First, we'll need to edit our services file:

sudo nano /etc/services

The following lines can go anywhere really, but the end of the file is convenient, particularly if you're planning to have many services running from this machine:

apt-cacher           3142/tcp
apt-cacher           3142/udp

3142 is the default port. I see no reason to change it, but if you have one, you may. If so, you'll need to change the daemon_port option in /etc/apt-cacher/apt-cacher.conf

One more file to edit:

sudo nano /etc/default/apt-cacher

Set AUTOSTART to 1 to get the daemon to launch at boot time.

To launch it now, simply type the following:

sudo /etc/init.d/apt-cacher start

Now, we're all done for the server side of things. Skip down to Part Two

Inetd Mode
To kick things off, we'll need to install the service:

sudo apt-get install inetutils-inetd

Next, we edit a couple of files:

sudo nano /etc/services

Add these two lines to the end of the file:

apt-cacher           3142/tcp
apt-cacher           3142/udp

Now, we have to setup inetd:

sudo nano /etc/inetd.conf

Add this line:

apt-cacher stream tcp nowait www-data /usr/sbin/tcpd /usr/sbin/apt-cacher -i

Now, restart the inetd server:

sudo /etc/init.d/inetutils-inetd restart

You're done!

Part Two: The Client
To use apt-cacher, your clients need to redirect their requests to the server machine. APT has its own proxy configuration. Thanks to omegamormegil for this tip
Type the following in a terminal:

echo 'Acquire::http::Proxy "http://hostname:3142";' | sudo tee /etc/apt/apt.conf.d/01proxy

Where hostname is a valid identifier for your server (for home users, an IP address will probably be convenient).

For laptop users, this means that you'll be unable to update using this method while on the road (unless you have a global route to your server, although the speed will be greatly reduced). To switch back to direct updates, type the following command:

echo 'Acquire::http::Proxy "http://";' | sudo tee /etc/apt/apt.conf.d/01proxy

Then once you get home, the command given above will get you using shared updates again.

If you prefer, APT will look for the last line in the file. You can cut/paste them as needed, but the above commands can be easily written into a script to automate the process for you.

From ace214, the following command will tell apt-get to use direct fetching methods:

export http_proxy=http://

As far as I know, this will only work for apt-get until you end your session (by logging out, or exiting/closing the terminal). If you're not tied to the GUI, this may serve as a useful solution for you.

Now, update!

sudo apt-get update

If all goes well, you should download packages as usual. Try downloading some software, or getting some updates. It should work pretty much as before.

If you are upgrading, these steps will not work. This appears to be a bug in Update Manager.

To upgrade, you must edit the 01proxy file to use direct updates as described above. Now, run the following command:

sudo sed --in-place s_http://_http://hostname:3142/_ /etc/apt/sources.list

The syntax may seem a little daunting, but this simply replaces any occurance of http:// with http://hostname:3142/. Once again, hostname must be a valid identifier for your server.

Upgrades should now work normally through the Update Manager. Once you've finished the upgrade, run the following command:

sudo sed --in-place s_http://hostname:3142/_http://_ /etc/apt/sources.list

Now edit the 01proxy file to set yourself to shared updates as described above.

So far, this is the only workaround I know of. I apologise to anyone who may have suffered this bug. Once more, thank you mssever for bringing this to my attention and providing a fix.

Part Three: The Transition
Chances are your server's up to date,and has a fair package cache of its own. Why duplicate all of this fine work? Apt-cacher comes with a migration helper for pre-caching. Type the following in a terminal to copy your cache over:

sudo cp /var/cache/apt/archives/*.deb /var/cache/apt-cacher/import
sudo /usr/share/apt-cacher/apt-cacher-import.pl

At the moment, the server machine itself will go downloading its own packages, rather than reusing those in the cache. To fix this, use the same proxy setup as used above on the clients, using "localhost" as the hostname.

The final step is optional, but it will help you free up space. The server is currently maintaining two caches: the regular apt one, and the shared cache.

NOTE: If you omit this step, APT will keep building its own cache. This may be of benefit to you when you uninstall apt-cacher if have limited bandwidth, but APT's operation will not be affected.

If your server uses Synaptic at all, there's an option under Settings -> Preferences -> Files to delete packages after installation. Thanks to Skip Da Shu for pointing this out to me.

This will only apply (in Ubuntu and Xubuntu) to packages fetched through the GUI. It will not work in Kubuntu, and will not work from the command line. For a more universal solution, enter the following command:

echo 'DPkg::Post-Invoke {"/bin/rm -f /var/cache/apt/archives/*.deb || true";};' | sudo tee /etc/apt/apt.conf.d/clean

This command is carried out just before APT quits, and will clean the cache of any .deb files. Thanks to bugmenot_user for this command.

Part Four: The Removal
To undo the changes made in this guide, you must remove the proxy configuration lines. You must also remove the file /etc/apt/apt.conf.d/clean, or set the option in Synaptic back to its original value.

Purging inetutils-inetd, inetutils-tcpd, and apt-cacher will undo most of the remaining changes. Remove the two lines from the end of /etc/services, and your system should be back to the way it was.

[omegamormegil]

Thanks for the info! I have apt-cacher running on my network, now. I figured out how to use a http proxy to make the clients update through the apt-cacher server, which might be easier than editing the sources.list file. It is MUCH easier, IMO, if you are on a laptop. I added the method I'm using to the end of this page:

https://help.ubuntu.com/community/Apt-Cacher-Server

Hope it's helpful!

[coolen]

Thanks for that. Does seem easier, especially when you've got multiple sources files.

[coolen]

Seems that your method runs a fair bit faster, too.

[bryceman]

Wait, which was faster? The inetd proxy or the linked apache2 proxy?

[spockrock]

Ok sorry now if I put apt-cacher on my 'server' and then use a proxy for all my desktop clients, does this mean only cached packages are downloaded on the clients??

[coolen]

To bryceman, I haven't tried the apache2 method, I'm afraid. I don't currently have any need for a web server, and it seems more straightforward to me to set this up on its own. I'll add it in the future for those interested, or if anyone's interested in contributing, I can get it up sooner.
When I mentioned that in the post, I was referring to the old proxy method, in which you had to edit each line of your sources list. My download speeds were generally about 10-20% of the speed of a direct download, and are usually running pretty close to normal speeds using APT's proxy configuration.

To spockrock, this does not in any way limit your experience. You can still download any package you want, from any repository you want. Once you have this set up, you shouldn't need to think about it unless you're on a laptop moving between networks.
What happens is that the request goes to the server machine, where the apt-cacher service will pick it up. It'll download the package, streaming a copy of it to you (the client), and keeping one for itself. If another client requests the same package, the service will simply send them a copy of the cached package.

[spockrock]

ok cool so its fine then the servers sources.lst and clients sources.lst files are different, as well if the client upgrades or installs a package the server machine will then use the cached package as well right?

[coolen]

Yep. The next time the server requests the new package (uhh, from itself), it'll send itself the cached one without touching the 'net.

[onegreenparker]

All sounds good, but will this work if the server is running 6.06 and the desktops are running anything later?