HOWTO: Download Ubuntu ISOs from multiple mirrors at once

[Josh1]

Please don't do something like this on release day... it might make your download speed look wonderful but puts unnecessary stress on the servers . Even if you spread your load on multiple mirrors, you're likely preventing someone else from getting their hands on a copy of Ubuntu.

If 50,000 people want to download ubuntu right now, and each use 10 connections, that will easily clog the server down.
But since there is loads of mirrors for ubuntu (there is even one that runs on WAIX, lol) you could put one connection per mirror..

[antini]

As always, Metalinks are available for new versions of Ubuntu at http://download.packages.ro/metalink/ubuntu/ as soon as they hit the mirrors.

The newly released Metalink tools (command line, http://prog.infosnel.nl/metalinks/) can be used to generate Metalinks. It scans files and adds their checksums and mirrors to a Metalink.

[antini]

Linux.com has a good article explaining Metalinks at titled Downloading bliss with Metalink:

Getting popular software off the Internet can sometimes be a struggle, even with all the mirrors and BitTorrent Samaritans out there. When the Fedora project released Fedora Core 6 last month, for instance, even several dozen mirrors weren't enough to serve everyone, and torrent speeds weren't good enough because of a scarcity of seeders. But thanks to Metalink I was able to sleep while my FC6 ISOs were downloading.

Metalink is an open standard that claims to make downloading easier, faster, and more reliable by helping users extract the last drop of juice out of their connection. But Metalink isn't your run-of-the-mill download accelerator. It is in fact a framework for use by other download clients, and bundles traditional HTTP and FTP methods of downloading files along with BitTorrent.

The Metalink standard replaces static URLs with a .metalink file, which is a simple XML file. The file in turn contains locations of all the mirrors of the application you want to download. In addition to HTTP and FTP mirrors and rsync, Metalink supports several P2P methods as well, including BitTorrent, ed2k, and magnet links. For example, the OpenOffice.org metalink contains links to more than 50 HTTP and FTP servers and a torrent.

If I were using a regular hyperlink to download OpenOffice.org and the server went down midway through the transfer, I would be left with an incomplete transfer that could either be resumed or not depending on the download client. But with Metalink, if one server goes down, the client software simply jumps to another mirror and resumes downloading from that point. In effect, this increases reliability, as all listed servers would need to be down for the file to be unavailable.

Files downloaded with metalinks are automatically verified. Metalink supports both MD5SUM and SHA1SUM checksums as well as PGP signatures, which are embedded in the .metalink file itself. Checksums are unique for every file, like a fingerprint. If there's an error in transfer, or if someone has maliciously replaced a good file with a bad one, the checksums won't match. If that happened with another file transfer method, you'd have to download the file again from another mirror. With Metalink, if the application you are downloading has a torrent, Metalink can use a torrent's partial file or chunk checksums to verify mirror downloads as well. If only a small chunk of a download has errors, Metalink just re-downloads that part instead of the whole file.

[Ziox]

What is the advantage of metalink over bittorrent? and what is a good program to use for ubuntu?

[antini]

What is the advantage of metalink over bittorrent? and what is a good program to use for ubuntu?

Metalink can use bittorrent, to supplement mirror downloads. In some places, p2p can't be used - like schools, businesses, and some ISPs. In places like that, regular downloads aren't blocked, so Metalink clients will skip p2p and use mirrors. There are also other features like OS, language, and location download filtering. Depending on the mirrors, in normal use Metalink downloads will be much faster.

aria2 is in the feisty repos. 'sudo apt-get install aria2' should work.

You might be able to download the source and compile it for edgy.
Something like the following should work:

sudo -s
echo 'deb-src http://archive.ubuntu.com/ubuntu feisty main restricted universe multiverse' >> /etc/apt/sources.list
apt-get update
cd /tmp
apt-get source aria2
apt-get build-dep aria2
cd aria2-*
debuild -uc -us
cd ..
dpkg -i aria2*.deb
rm -rf aria2*
exit

[elektronaut]

In case you get 'command not found' for 'debuild', it's in the devscripts package:

sudo aptitude install devscripts

[antini]

Thanks for the help.

Metalinks contributed to the distribution of openSUSE 10.2...

[smartbei]

I am hacing a problem using aria2/metalinks. the download proceeds well, and for some reason this is the only way i can get speeds that are practical for downloading a dvd iso. On torrents I get really low speeds (max 5KB/2, avg 1.5KB/s)...

when the download finishes it checks the file and always says 'checksum failed'. Still, i have done this twice now, and each time independantly verified that the md5sum doesn't match up. Still, it takes a long time to download and I wonder how much the fact that it doesn't match actually has to do with anything...

I am downloading a DVD iso for SUSE (it is supposed to work better on my hardware) - what do you suggest?

[antini]

I am hacing a problem using aria2/metalinks. the download proceeds well, and for some reason this is the only way i can get speeds that are practical for downloading a dvd iso. On torrents I get really low speeds (max 5KB/2, avg 1.5KB/s)...

when the download finishes it checks the file and always says 'checksum failed'. Still, i have done this twice now, and each time independantly verified that the md5sum doesn't match up. Still, it takes a long time to download and I wonder how much the fact that it doesn't match actually has to do with anything...

I am downloading a DVD iso for SUSE (it is supposed to work better on my hardware) - what do you
suggest?

My guess is that you are getting errors in the transfer due to errors in your connection or one of the mirrors is serving up bad data (more likely the first).

The current version of aria2 doesn't support the advanced error correction that metalink allows (just the speed).

Since you have already downloaded the ISO, there are probably just a few small errors in the whole file, which causes the checksum to fail. You can use a torrent and another torrent client (don't use aria2) and it will scan the file and fix errors. Or you can use rsync, see the bottom of http://en.opensuse.org/Metalinks

This presupposes that you were downloading openSUSE-10.2-GM-DVD-i386.iso via metalink, and at some point (probably the end) you got an error. Use rsync like this to download any missing chunks:

cd /directory/ofthe/iso
rsync -avvP rsync://mirrors.kernel.org/opensuse/distribution/10.2/iso/dvd/openSUSE-10.2-GM-DVD-i386.iso .

Note: we can just use a dot (".") at the end here since we simply want to continue the ISO with the same filename, in the current directory.

I know these extra steps are annoying, sorry. Hopefully the next version of aria2 supports the features that will automate this.

[smartbei]

Thanks for the quick reply. I tried using the said rsync package but I keep getting the following error for the mirror they used:

$ rsync -avvP rsync://mirrors.kernel.org/opensuse/distribution/10.2/iso/dvd/openSUSE-10.2-GM-DVD-i386.iso .
opening tcp connection to mirrors.kernel.org port 873
rsync: connection unexpectedly closed (0 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(463) [receiver=2.6.8]

And a different one for another I tried:

$ rsync -avvP rsync://sunsite.informatik.rwth-aachen.de/pub/linux/opensuse/distribution/10.2/iso/dvd/openSUSE-10.2-GM-DVD-i386.iso .
opening tcp connection to sunsite.informatik.rwth-aachen.de port 873
rsync: failed to connect to sunsite.informatik.rwth-aachen.de: Connection refused (111)
rsync error: error in socket IO (code 10) at clientserver.c(107) [receiver=2.6.8]

I checked the router and I think I have it set up so port 873 is open - though I am not sure as to how to test this.

I also tried bittorent and opened the torrent in Azureus, which, after checking, said that the file is 77.9% complete, explaining the wrong md5sum. However, because my download speed is so low, that would still take forever to make up with azureus. Any ideas what the error is?

EDIT: I made my computer a temporary DeMilitarized Zone for testing and I see that rsync works now. Apparently I was still blocking a neccessary port.