antini
May 26th, 2010, 02:13 AM
the author of aria2 (http://aria2.sourceforge.net) was inspired by something that used another downloader to speed up apt-get.
First, if you don't already have aria2 installed,
sudo apt-get install aria2
aria2 gets the URI, filename, size, sha-256 hash from apt-get, creates a metalink (http://www.metalinker.org), & downloads the files w/ aria2. the nice thing about aria2 is that it can get parts of files from different mirrors, but I don't know if this does that yet. it's already a couple times faster than regular apt-get.
he writes (https://sourceforge.net/apps/phpbb/aria2/viewtopic.php?f=1&t=52&start=0):
apt-get -y --print-uris -qq' outputs needed package metainfo: URI, filename, size, sha-256 hash.
We can transform this output to just list of URIs and feed it to aria2.
aria2 can download multiple files at one so it is faster than axel at this point.
But wait, we have file size and sha-256 hash. Why not convert the apt-get output to metalink for more reliable download?
I wrote a simple Python script called apt2metalink.py to convert apt-get output to metalink4. Please use python2.6.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
import re
if __name__ == "__main__":
entries = []
while True:
line = sys.stdin.readline()
if not line: break
uri, name, size, chksum = line.split()
uri = re.sub(r"^'|'$", "", uri)
hashfunc, hashvalue = chksum.split(':')
hashfunc = hashfunc.lower()
if hashfunc == 'sha256':
hashfunc = 'sha-256'
entries.append((uri, name, size, hashfunc, hashvalue))
print """<?xml version="1.0" encoding="UTF-8"?>
<metalink xmlns="urn:ietf:params:xml:ns:metalink">"""
for e in entries:
print """<file name="{name}">
<size>{size}</size>
<hash type="{hashfunc}">{hashvalue}</hash>
<url priority="1">{uri}</url>
</file>""".format(uri=e[0],name=e[1], size=e[2], hashfunc=e[3], hashvalue=e[4])
print """</metalink>"""
Now aria2 can validate checksums!
Let's take a look of actual usage example. First, save above script as 'apt2metalink.py'.
To get packages for upgrade, we can use following command chain:
apt-get -y --print-uris -qq upgrade|python2.6 apt2metalink.py|aria2c -d /var/cache/apt/archives/ -M- --file-allocation=none
After successful downloads, you can issue 'apt-get upgrade' to install packages.
You can replace 'upgrade' with list of packages to install or upgrade.
NOTE: I had to split the above in two like so:
apt-get -y --print-uris -qq upgrade|python2.6 apt2metalink.py > tmp.meta4
sudo aria2c -d /var/cache/apt/archives/ -M tmp.meta4 --file-allocation=none
First, if you don't already have aria2 installed,
sudo apt-get install aria2
aria2 gets the URI, filename, size, sha-256 hash from apt-get, creates a metalink (http://www.metalinker.org), & downloads the files w/ aria2. the nice thing about aria2 is that it can get parts of files from different mirrors, but I don't know if this does that yet. it's already a couple times faster than regular apt-get.
he writes (https://sourceforge.net/apps/phpbb/aria2/viewtopic.php?f=1&t=52&start=0):
apt-get -y --print-uris -qq' outputs needed package metainfo: URI, filename, size, sha-256 hash.
We can transform this output to just list of URIs and feed it to aria2.
aria2 can download multiple files at one so it is faster than axel at this point.
But wait, we have file size and sha-256 hash. Why not convert the apt-get output to metalink for more reliable download?
I wrote a simple Python script called apt2metalink.py to convert apt-get output to metalink4. Please use python2.6.
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import sys
import re
if __name__ == "__main__":
entries = []
while True:
line = sys.stdin.readline()
if not line: break
uri, name, size, chksum = line.split()
uri = re.sub(r"^'|'$", "", uri)
hashfunc, hashvalue = chksum.split(':')
hashfunc = hashfunc.lower()
if hashfunc == 'sha256':
hashfunc = 'sha-256'
entries.append((uri, name, size, hashfunc, hashvalue))
print """<?xml version="1.0" encoding="UTF-8"?>
<metalink xmlns="urn:ietf:params:xml:ns:metalink">"""
for e in entries:
print """<file name="{name}">
<size>{size}</size>
<hash type="{hashfunc}">{hashvalue}</hash>
<url priority="1">{uri}</url>
</file>""".format(uri=e[0],name=e[1], size=e[2], hashfunc=e[3], hashvalue=e[4])
print """</metalink>"""
Now aria2 can validate checksums!
Let's take a look of actual usage example. First, save above script as 'apt2metalink.py'.
To get packages for upgrade, we can use following command chain:
apt-get -y --print-uris -qq upgrade|python2.6 apt2metalink.py|aria2c -d /var/cache/apt/archives/ -M- --file-allocation=none
After successful downloads, you can issue 'apt-get upgrade' to install packages.
You can replace 'upgrade' with list of packages to install or upgrade.
NOTE: I had to split the above in two like so:
apt-get -y --print-uris -qq upgrade|python2.6 apt2metalink.py > tmp.meta4
sudo aria2c -d /var/cache/apt/archives/ -M tmp.meta4 --file-allocation=none