Bots and internet search engines

sunama

sunama

sunama

Soldato
Joined
18 Oct 2002
Posts
15,861
Location
NW London
According to Google's terms of service, they do not allow programs (or bots), to use their search engine to search for stuff.

Does anybody know if there are any search engines out there, which do allow bots to to use their search engines? Do Bing and Yahoo allow bot searches or are there any open source, less well-known search engines which can be used by bots?
 
I doubt it. Are you doing SEO?

I think you can do it with a special commercial license from Google but it probably costs tens of thousands of dollars.
 
Not SEO.

I'm creating a program which needs to type in search terms into a search engine and retrieve the number of search results.

I can pretty much use any search engine (it doesn't have to be google), so I'm looking for a search engine which permits a program (non-human), to use the search engine.

Google's own terms of use policy stipulates the prohibition of using automated programs (or bots). However, I have looked up Bing's and Yahoo's terms of use policies and neither give any such stipulations and do not make any mention regarding bots.
 
Are you sure this isn't allowed by Google? I would think they've said that to cover their backs with regards to people knocking up bots that do thousands of searches per second/minute.

As long as you're doing something reasonable with it then I can't see why they wouldn't allow it.
 
slylittlefox said:
Are you sure this isn't allowed by Google? I would think they've said that to cover their backs with regards to people knocking up bots that do thousands of searches per second/minute.

As long as you're doing something reasonable with it then I can't see why they wouldn't allow it.
Click to expand...

This is exactly what I was thinking. What they are trying to avoid is 1000s of people using 1000s of bots hammering their search engines and slowing their search engine down. This technique could also be used maliciously, just to slow down (or break) their servers.

However, I want to cover my back, before creating a program which uses their search engine. The best way to avoid any future wrangles is to find out about any other search engines which can be used.

I don't really need Google for the searches, but as their search engine is so good, it would be a shame not to use it.
 
How are you planning on using this program? If the user types in a search term and it returns the number of search results then that's not a bot or an automated program.
 
kwerk said:
Google will ban your IP address if you do too many searches too quick.
Click to expand...

Yep, this happened to me a couple of years ago when I was doing some SEO analysis. Blocked for 24 hours. Ended up using a some tool that mimicked human searches, can't remember what it was called now but it worked well enough.
 
The reason they don't want bots is because people use them to Google Bomb. If something is entered into Google millions of times, then it gets ranked higher. It's not to do with DDoS or anything like that, purely on results.

For example if someone bombed Google with "justin bieber is gay" enough times successfully, then anyone searching for "justin bieber" will get the search suggestion of "justin bieber is gay" and the results will prioritise for that instead of "justin bieber".
 
Dj_Jestar said:
The reason they don't want bots is because people use them to Google Bomb. If something is entered into Google millions of times, then it gets ranked higher. It's not to do with DDoS or anything like that, purely on results.

For example if someone bombed Google with "justin bieber is gay" enough times successfully, then anyone searching for "justin bieber" will get the search suggestion of "justin bieber is gay" and the results will prioritise for that instead of "justin bieber".
Click to expand...

I understand.

But are there any other search engines, which are less strict?

In fairness, it doesnt have to be google which I use. I just need a basic search engine to return the search result count....thats it. I dont want any comebacks at a later date from google (or any other search engine). The lesser known search engine is less likely to have strict terms of use policies.

I won't be hammering the search engine with the same search term. My algorithm inputs the same search phrase about 2-6 times...no more. The same search term will be entered into the search engine with a time gap of at least 2 weeks.
 
kwerk said:
Google will ban your IP address if you do too many searches too quick.
Click to expand...

Hehehe.

During testing, I hammered google with the same search term and I also received the 24 hour ban. The testing is now complete, so I won't need to hammer google with the same search term again.
 
Skyfall said:
How are you planning on using this program? If the user types in a search term and it returns the number of search results then that's not a bot or an automated program.
Click to expand...

I'm creating an AI program. The first stage of this is to a create a Natural Language Processor component.

As part of this, a user will type out some information, in natural language and the AI may decide to check this information out using a search engine.

The AI will eventually have full capabilities to research topics, so will have major search engine usage capabilities.

The AI program will be that which creates a search phrase and enters it into the search box...not the user.

In essence, I want to avoid problems further down the line, so want to sort out a reliable search engine now, rather than later, so that I won't have to make major changes at a later date.

Details are in my signature.
 
You must log in or register to reply here.
Back
Top Bottom