PDA

View Full Version : Google spiders?



senectus
November 24th, 2004, 06:15 AM
I was poking around the forums looking for something to entertain me (yeah I'm pretty bored atm) when I noticed that there was a LOT of users called "Google Spider" browsing the forums.. (see attatchment)

Now I figure this must be the google web crawlers that go out and index the web for search engine purposes..

Then I thought.. Jeez that must put a heafty load on the forum resources..

Would that be correct? does having dozens and dozens of these hits, place a large load on the forums?

jdodson
November 24th, 2004, 04:10 PM
it allows google to bring up search matches if someone is having a problem and types it into google. this brings traffic to the ubuntu forums site, this could bring more people to the distro. plus its not that much traffic fwiw in the long run anyway. the google algorythms are sensitive to not overloading any webserver, if slashdot continually linked to our site, then we might have something to worry about :mrgreen:

plasmo
November 24th, 2004, 04:30 PM
a simple robots.txt should fix that but hey
we want more hits!1 8-)