PDA

View Full Version : Multi-threaded downloads in Linux ?



mips
September 27th, 2006, 09:52 PM
I never seem to get good download speeds in ubuntu. Using D4X or WGET simply does not give me the same performance as say GetRight in windows.

With getright i have 5 simultanious segmented downloads of the same files. it literally maes out my adsl connection (384kb/s) to the point that i even battle to just browse the net.

In ubuntu it does not really get better than say 200kb/s.

If I download ISO's i use windows as i find linux to slow.

How can i improve this ???

croak77
September 27th, 2006, 09:58 PM
aria2

http://aria2.sourceforge.net/



aria2 has segmented downloading engine in its core. It can download one file from multiple URLs or multiple connections from one URL. This results in very high speed downloading, very much faster than ordinary browsers.
As of 0.3.0 release, It can also download BitTorrent files.
We implemented this engine in single-thread model. The architecture is very clean and easy to extend.

Unlike Aria, which has GTK+ interface, aria2 provides command-line interface only. But GUI-lessness brings lower resource requirement. The physical memory usage is typically 3MB(normal HTTP/FTP downloads) to 5MB(BitTorrent downloads).
As of 0.7.1 release, aria2 supports asynchronous DNS using c-ares or ares. This can improve segmented download performance, especially in Metalink download.

Command-line interface
Download files through HTTP/HTTPS/FTP/BitTorrent
HTTP Proxy support
FTP through HTTP Proxy
HTTP BASIC authentication support
HTTP Proxy authentication support
Segmented downloading
Cookie support(currently aria2 ignores "expires")
Run as a daemon process.
Selective download in multi-file torrent
BitTorrent Fast extension support
Metalink version 3.0 support(HTTP/FTP/BitTorrent)

mips
September 27th, 2006, 10:26 PM
Thx. Unfortunately it is cli based. While looking at it I also came across:

http://dfast.sourceforge.net/

I'll give them both a try.

croak77
September 27th, 2006, 10:46 PM
Thx. Unfortunately it is cli based. While looking at it I also came across:



You mean thank god it's cli based ;)

kpkeerthi
September 27th, 2006, 10:57 PM
If you like firefox use the DownThemAll! extension. I'm using it and can't be any happier.
https://addons.mozilla.org/firefox/201/

I tried d4x but somehow felt very cartoonish to me.

user1397
September 28th, 2006, 12:28 AM
thats weird. using bittorrent, i get download speeds of like max 450 kbps in windows and linux.

ember
September 28th, 2006, 12:32 AM
I agree with kpkeerthi, DownThemAll is very useful for downloading - I use it both in Windows and Linux.

richbarna
September 28th, 2006, 12:37 AM
I never seem to get good download speeds in ubuntu. Using D4X or WGET simply does not give me the same performance as say GetRight in windows.

With getright i have 5 simultanious segmented downloads of the same files. it literally maes out my adsl connection (384kb/s) to the point that i even battle to just browse the net.

In ubuntu it does not really get better than say 200kb/s.

If I download ISO's i use windows as i find linux to slow.

How can i improve this ???

If the isos are available as a torrent download, I use ktorrent.

mips
September 30th, 2006, 09:17 PM
I tried compiling from source, but I must be doing something wrong or missing dependancies.

Anybody got a deb ???

I see you can plug this into other download utils which is cool.




aria2

http://aria2.sourceforge.net/



aria2 has segmented downloading engine in its core. It can download one file from multiple URLs or multiple connections from one URL. This results in very high speed downloading, very much faster than ordinary browsers.
As of 0.3.0 release, It can also download BitTorrent files.
We implemented this engine in single-thread model. The architecture is very clean and easy to extend.

Unlike Aria, which has GTK+ interface, aria2 provides command-line interface only. But GUI-lessness brings lower resource requirement. The physical memory usage is typically 3MB(normal HTTP/FTP downloads) to 5MB(BitTorrent downloads).
As of 0.7.1 release, aria2 supports asynchronous DNS using c-ares or ares. This can improve segmented download performance, especially in Metalink download.

Command-line interface
Download files through HTTP/HTTPS/FTP/BitTorrent
HTTP Proxy support
FTP through HTTP Proxy
HTTP BASIC authentication support
HTTP Proxy authentication support
Segmented downloading
Cookie support(currently aria2 ignores "expires")
Run as a daemon process.
Selective download in multi-file torrent
BitTorrent Fast extension support
Metalink version 3.0 support(HTTP/FTP/BitTorrent)

skymt
September 30th, 2006, 10:14 PM
I tried compiling from source, but I must be doing something wrong or missing dependancies.

Anybody got a deb ???

I see you can plug this into other download utils which is cool.

What errors do you get when you try to compile?

croak77
September 30th, 2006, 10:33 PM
Aria2 is in Raphink's unofficial packages for Ubuntu

http://www.raphink.net/ubuntu/

Josh1
September 30th, 2006, 10:39 PM
I never seem to get good download speeds in ubuntu. Using D4X or WGET simply does not give me the same performance as say GetRight in windows.

With getright i have 5 simultanious segmented downloads of the same files. it literally maes out my adsl connection (384kb/s) to the point that i even battle to just browse the net.

In ubuntu it does not really get better than say 200kb/s.

If I download ISO's i use windows as i find linux to slow.

How can i improve this ???
Prozilla. Works just like reget for windows.

mips
September 30th, 2006, 11:09 PM
Aria2 is in Raphink's unofficial packages for Ubuntu

http://www.raphink.net/ubuntu/

Thx, they are not the latest but I'll have a look.

mips
September 30th, 2006, 11:10 PM
What errors do you get when you try to compile?

I'll post the output of ./config & make tomorrow.

mips
October 1st, 2006, 03:32 PM
Attached find output of ./configure & make...

slimdog360
October 1st, 2006, 03:41 PM
there is also kget, you can make it the default dl manager in konqueror.

mips
October 1st, 2006, 05:48 PM
there is also kget, you can make it the default dl manager in konqueror.

As far aas i know kget does not support segmented downloading. I have used it before though.

croak77
October 1st, 2006, 06:26 PM
Attached find output of ./configure & make...

You need libxml 2.6.26 or greater.

mips
October 1st, 2006, 06:41 PM
You need libxml 2.6.26 or greater.

I have 2.6.24. Where would I find 2.6.26 ?

skymt
October 1st, 2006, 08:47 PM
I have 2.6.24. Where would I find 2.6.26 ?

Edgy has .26. If you don't want to upgrade yet, your best option is probably to compile it yourself (http://xmlsoft.org/).

mips
October 1st, 2006, 09:08 PM
Edgy has .26. If you don't want to upgrade yet, your best option is probably to compile it yourself (http://xmlsoft.org/).

I think I'll wait for edgy then or try the older deb in the meantime.

mips
October 17th, 2006, 08:32 PM
wxDownload Fast does the job just fine for me so far.

I also have aria2 installed but have not used it much so cannot comment right now.

Where did I get them from ? http://www.getdeb.net/

antini
December 11th, 2006, 02:46 AM
aria2 (http://aria2.sourceforge.net) is in feisty. There's .debs for wxDownload Fast (http://dfast.sourceforge.net) on its download page too.

The new version of KGet in KDE4 will support segmented downloading as well.

Also, check out Metalink (http://www.metalinker.org) which is a file used by download managers just for this sort of thing.

antini
July 23rd, 2007, 06:59 AM
DownThemAll! (http://www.downthemall.net/main/install-it/downthemall-10-beta/) is a Firefox extension that does really fast multithreaded downloads.

kvonb
July 23rd, 2007, 07:18 AM
wxDownload Fast does the job just fine for me so far.

I also have aria2 installed but have not used it much so cannot comment right now.

Where did I get them from ? http://www.getdeb.net/

Hey, thanks for that link to GetDeb mips, really nice stuff :)

mips
July 23rd, 2007, 08:21 AM
Hey, thanks for that link to GetDeb mips, really nice stuff :)

No problem ;) Where have you been as that site has been around for a while.

kvonb
July 23rd, 2007, 10:24 AM
No problem ;) Where have you been as that site has been around for a while.

The older I get, the less I see, or want to see :D