PDA

View Full Version : Scroogle may be kaput! Which is most unfortunate...



handy
May 11th, 2010, 09:16 AM
We regret to announce that our Google scraper may have to be permanently retired, thanks to a change at Google. It depends on whether Google is willing to restore the simple interface that we've been scraping since Scroogle started five years ago. Actually, we've been using that interface for scraping since Google-Watch.org began in 2002.

This interface (here's a sample (http://www.scroogle.org/simple.html) from years ago) was remarkably stable all that time. During those eight years there were only about five changes that required some programming adjustments. Also, this interface was available at every Google data center in exactly the same form, which allowed us to use 700 IP addresses for Google.

That interface was at www.google.com/ie but on May 10, 2010 they took it down and inserted a redirect to /toolbar/ie8/sidebar.html. It used to have a search box, and the results it showed were generic during that entire time. It didn't show the snippets unless you moused-over the links it produced (they were there for our program, so that was okay), and it has never had any ads. Our impression was that these results were from Google's basic algorithms, and that extra features and ads were added on top of these generic results. Three years ago Google launched "Universal Search," which meant that they added results from other Google services on their pages. But this simple interface we were using was not affected at all.

Now that interface is gone. It is not possible to continue Scroogle unless we have a simple interface that is stable. Google's main consumer-oriented interface that they want everyone to use is too complex, and changes too frequently, to make our scraping operation possible.

Over the next few days we will attempt to contact Google and determine whether the old interface is gone as a matter of policy at Google, or if they simply have it hidden somewhere and will tell us where it is so that we can continue to use it.

Thank you for your support during these past five years. Check back in a week or so; if we don't hear from Google by next week, I think we can all assume that Google would rather have no Scroogle, and no privacy for searchers, at all.

Daniel Brandt, Public Information Research, scroogle AT lavabit.com


P.S. Many thanks for all the support emails. A word to those who are emailing us with links to simplified Google pages with just the initial search entry box on them: Sorry, but this doesn't help us. It's the simplicity of the pages with search results that matter. Scroogle uses it's own gateway pages, but we parse out the links from Google's results page before we pass them back to you.

See the "sample" link in the second paragraph of this page, where you have to mouse-over each link to see the snippet. Results like this were not too difficult to parse. More importantly, they were stable. The query to Google that produced results like this is illustrated by the URL behind the "Next" link at the bottom of that sample (which no longer works, obviously). The parameters in URLs like that could be manipulated for language selection and number of results per page, just like for all Google searches.

Whether Google cares to restore this simple interface is a question of fundamental corporate policy. Frankly, we've always felt that we'd reach this point with Google sooner or later. Years ago it was Microsoft's decision in a different context, before web search engines even existed:

handy
May 11th, 2010, 01:31 PM
Added the recent P.S. to the OP.

Sporkman
May 11th, 2010, 02:56 PM
Why doesn't Scroogle just build its own search engine?

Trail
May 11th, 2010, 04:41 PM
Why doesn't Scroogle just build its own search engine?

I heard they were getting some compiler errors. Undifined reference to #ALOTOFMONEY.

Yeah, i know it's linking, whatever)

Sporkman
May 11th, 2010, 04:48 PM
I heard they were getting some compiler errors. Undifined reference to #ALOTOFMONEY.

Yeah, i know it's linking, whatever)

If they really believed in what they're doing, they'd mow some lawns / ask for donations / sell add space / etc. Besides, they could keep costs low by using free open source software.

handy
May 12th, 2010, 01:10 AM
If they really believed in what they're doing, they'd mow some lawns / ask for donations / sell add space / etc. Besides, they could keep costs low by using free open source software.

I usually respect what you have to say.

Sporkman
May 12th, 2010, 01:15 AM
I usually respect what you have to say.

Ouch! :lol:

Sorry but I have hard time having sympathy for a scraper site. And I was only half joking about them making their own search engine - correct me if I'm wrong, but there are open source frameworks for web spidering/indexing/etc, no? They certainly wouldn't be able to implement anything on the scale of Google, but they could at least present a small-scale alternative that is open & private.

handy
May 12th, 2010, 04:10 AM
Scroogle is back! :D

ubunterooster
May 12th, 2010, 04:14 AM
Do you, uh, Sroo-oooogle?

handy
May 12th, 2010, 04:54 AM
Do you, uh, Sroo-oooogle?

Multiple times a day.

Have been doing it for years...

ubunterooster
May 12th, 2010, 05:35 AM
ever since I saw Scroogle, I liked it. It has the results of Googlee in a simple, uncluttered, interface. As for privacy, I try to keep a safe distance from certain sites and services that take to much of my info. Yes, that includes Google

Chronon
May 12th, 2010, 05:55 AM
If Scroogle stops scraping can't people just use TOR/proxies?

handy
May 12th, 2010, 07:31 AM
If Scroogle stops scraping can't people just use TOR/proxies?

Tor slows down the users internet experience.

It would seem that Google has reacted positively to Scroogle's request. But I have no documentary proof of that.

The GoogleSharing (http://www.googlesharing.net/) Firefox add-on is a good thing to use whether you are using Scroogle or not.

Chronon
May 12th, 2010, 08:01 AM
Tor slows down the users internet experience.

It would seem that Google has reacted positively to Scroogle's request. But I have no documentary proof of that.

The GoogleSharing (http://www.googlesharing.net/) Firefox add-on is a good thing to use whether you are using Scroogle or not.

True, but you can configure only specific traffic to go through TOR if you like.

t0p
May 12th, 2010, 10:00 AM
ever since I saw Scroogle, I liked it. It has the results of Googlee in a simple, uncluttered, interface. As for privacy, I try to keep a safe distance from certain sites and services that take to much of my info. Yes, that includes Google

Are you aware that, just because a service like Scroogle says it isn't harvesting your data, doesn't mean it's not harvesting your data?

Not that I think there's anything untoward about Scroogle. But it's something to bear in mind, especially if you have real security issues.

Tristam Green
May 12th, 2010, 02:09 PM
haha "just because you're paranoid doesn't mean they're not after you".

ubunterooster
May 12th, 2010, 02:56 PM
I use noscript + scroogle. I have TOR but it boggs stuff down. IMO, scroogle is a noble program that far surpasses most other search engines

RiceMonster
May 12th, 2010, 03:07 PM
Are you aware that, just because a service like Scroogle says it isn't harvesting your data, doesn't mean it's not harvesting your data?

Not that I think there's anything untoward about Scroogle. But it's something to bear in mind, especially if you have real security issues.

Exactly; people should not use the internet, because they're probably being spied on at all times.

ubunterooster
May 12th, 2010, 03:12 PM
Exactly; people should not use the internet, because they're probably being spied on at all times.
??

Tristam Green
May 12th, 2010, 03:21 PM
??

It's the only thing that can be inferred from t0p's post, really.

ubunterooster
May 12th, 2010, 04:09 PM
It's the only thing that can be inferred from t0p's post, really.
Ah, ok

NMFTM
May 12th, 2010, 04:40 PM
Besides privacy, I prefer Scroogle to Google because they highlight the individual words you've searched for instead of bolding them. Which would make it impossible to distungish between them.

aysiu
May 12th, 2010, 04:40 PM
No, I think the implication is that there is no reason to trust an organization simply on the grounds that it is against another organization you don't trust.

If you don't trust Google's privacy policy, why would you trust Scroogle's? What has Scroogle done to earn your trust?

ubunterooster
May 12th, 2010, 04:42 PM
No, I think the implication is that there is no reason to trust an organization simply on the grounds that it is against another organization you don't trust.

If you don't trust Google's privacy policy, why would you trust Scroogle's? What has Scroogle done to earn your trust?
Scroogle does not offer me cookies like Google does.

Every day Scroogle crumbles 300,000 cookies and blocks a million ads.
Help us make the web green again with a tax-deductible donation (http://www.scroogle.org/donatesc.html).

aysiu
May 12th, 2010, 04:51 PM
Scroogle does not offer me cookies like Google does.

Every day Scroogle crumbles 300,000 cookies and blocks a million ads.
Help us make the web green again with a tax-deductible donation (http://www.scroogle.org/donatesc.html).

Why do they have to offer you cookies?

ubunterooster
May 12th, 2010, 06:12 PM
um, Google offers cookies so they can get all the data for marketing (studies say google knows more about an average individual than does the NSA). Scroogle gives none of those tracking cookies

aysiu
May 12th, 2010, 06:15 PM
um, Google offers cookies so they can get all the data for marketing (studies say google knows more about an average individual than does the NSA). Scroogle gives none of those tracking cookies
They don't have to give cookies.

All your searches go through their servers, which they then pass on to Google. They say they don't save search terms and that they delete logs after 48 hours, but why should you believe them?

Tristam Green
May 12th, 2010, 06:20 PM
No, I think the implication is that there is no reason to trust an organization simply on the grounds that it is against another organization you don't trust.

If you don't trust Google's privacy policy, why would you trust Scroogle's? What has Scroogle done to earn your trust?

+1 to aysiu. It's almost like offering the option of "Pick One: [ ] Old and Nasty [ ] Shiney and New" without actually giving a picture of the item in question.


They don't have to give cookies.

All your searches go through their servers, which they then pass on to Google. They say they don't save search terms and that they delete logs after 48 hours, but why should you believe them?

Realistically, though, not all users are equipped with the knowhow to be an oversight committee. There does have to be some modicum of faith placed in a company to do good when one decides to go with that company/service provider.

aysiu
May 12th, 2010, 06:26 PM
Realistically, though, not all users are equipped with the knowhow to be an oversight committee. There does have to be some modicum of faith placed in a company to do good when one decides to go with that company/service provider. That's my point exactly. I don't have blind trust in Google. But I don't have blind trust in Scroogle either. I use plenty of services I don't fully trust to be looking out for my best interests. That's the only way to use the internet. You determine what you can trust best and reveal as much information as you're comfortable revealing. Not everything is under your control. Theoretically, you can trust your bank to not reveal your social security number (if you're an American--not sure what the personal identifier is for other countries) and birthdate, but all it takes is one rogue employee to leak it or some network administrator to not patch a security hole, and your information is out there.

You can't go through life blindly trusting everybody. But you also can't imagine there is some select list of companies you can blindly trust and be 100% safe (Scroogle doesn't get a free pass for trustworthiness just because they say they don't do something Google supposedly does). You just try to strike a balance and get a functional reasonable sense of privacy and security and go on with it.

ubunterooster
May 12th, 2010, 07:09 PM
Actually I think that unless there is something better, blind faith can be helpful.

My father is paranoid and thinks: his webcam is watching him, the computer is keylogged, Police pull people over near our house (1/2 mile away) just to watch him, our relatives would kill him if they could...

I prefer to trust until I see evidence not to. Otherwise we could live our entire lives cringing as someone walks by, thinking they might be trying to kill us. Blue pill?

aysiu
May 12th, 2010, 07:39 PM
Faith and blind faith are not the same thing.

Tristam Green
May 12th, 2010, 07:49 PM
Faith and blind faith are not the same thing.

ain't that (http://en.wikipedia.org/wiki/Faith_no_more) the truth! (http://en.wikipedia.org/wiki/Blind_faith)

ubunterooster
May 12th, 2010, 07:50 PM
Faith and blind faith are not the same thing.
True; just ignore every instance of "blind" that I used.

handy
May 13th, 2010, 02:39 AM
They don't have to give cookies.

All your searches go through their servers, which they then pass on to Google. They say they don't save search terms and that they delete logs after 48 hours, but why should you believe them?

Sometimes I do a little research, then evaluate what I have learned & then decide if I will trust what I have learned. I can't go & audit Scroogle's internal systems, but I do use them.

If I've been hoodwinked, then so be it.

Whereas I know that Google, Amazon & so many others on the net truly desire to know what sites I visit, for their profit.

I consider such tactics an invasion of my privacy, so I do what I can (without using a proxy service - apart from GoogleSharing - which also may or may not be trustworthy) to make it harder for these entities to profit from knowledge of what I do.

I would like there to exist an international law that made it an individual's choice as to whether they wanted to be tracked in any way on the internet or not.

granger99801
December 31st, 2010, 06:11 AM
... for their profit.
[edit]
I would like there to exist an international law that made it an individual's choice as to whether they wanted to be tracked in any way on the internet or not.

As long as we're making wishes, and bumping an old thread found inadvertently through a scroogled search...

When will I be paid per view for all these pop up sidebarring commercials???

Direct to consumer advertising? Neilsons Ratings sends me cashola buckazoid units in paper dinero for itemizing my viewing data matrix... If they're only kindly trying to sell me that which I want to buy, it should seem a small charge to ask for several debit units from each, for obliging them by being their marketing audience.

handy
January 1st, 2011, 12:43 AM
@granger: If it is opt-in, advertising is fine. (Tracking is not, collecting information about customers is not, unless those people choose for such to happen.)

I'm not going to tell other people how they should use the internet. I just don't accept others telling me how to use the internet. E.g. any site that blocks me due to my use of Adblock +, has just lost another viewer.