API's for search-engine visualization research

[bmeyer]

I'm conducting research with a professor here (Valparaiso University) developing 2D and 3D search-engine result visualizations. Up until now, we've been testing our systems on the Federal Code, which we have in its entirety on an external hard-drive.

However, this definitely needs to be expanded. Do any of you have any ideas of sites' APIs we could input search terms and receive a large output of results? Maybe along the lines of an encyclopedia database, news archive, search-engine, etc. We may look into downloading all of Wikipedia, but would like to stick to online APIs. Any ideas?

Google won't work as they limit the total search results to 100 and limit the total amount of searches you can do in a day...

[bmeyer]

*bump*

Any ideas? Thanks in advance...

[saulgoode]

Perhaps Archive.org?

[rabid9797]

dear bmeyer,

i give you the end all in photo searching: http://dearcomputer.nl/gir/ (also known as Google Image Ripper)

[bmeyer]

Any other ideas? I like the idea of archive.org, but after emailing them I found out that they currently have no APIs. They're working with an NSF sponsorship to develop researcher access -- the development is occurring at Cornell and isn't finished yet...