PDA

View Full Version : Wheres the best place to start a web crawler?


jimi_hendrix
March 13th, 2009, 01:05 PM
with nothing better to do this morning i made a basic web crawler to wake myself up...now where should i start it?

i just want to see how efficient it is
asimon
March 13th, 2009, 01:54 PM
with nothing better to do this morning i made a basic web crawler to wake myself up...now where should i start it?

i just want to see how efficient it is

Why not this forum?, they would be happy to meet another crawler. :popcorn:
koenn
March 13th, 2009, 03:30 PM
If by crawler you mean a program that follows hyperlinks, you probably want a starting point that leads to lots of links to lots of different places.

Maybe try a web directory ?
http://www.dmoz.org/
jimi_hendrix
March 13th, 2009, 05:58 PM
If by crawler you mean a program that follows hyperlinks, you probably want a starting point that leads to lots of links to lots of different places.

Maybe try a web directory ?
http://www.dmoz.org/

yes i ment that