googlebot banned from these forums?

[blade007]

I was curious about forum running so I checked the robots.txt on these forums. Guess what ... practically everything is disallowed from googlebot.

What's the idea in that then? To stop people using OCUK as a link, or to stop Googlebot giving Spie a bigger bandwidth bill?

 

[Zefan]

to stop Googlebot giving Spie a bigger bandwidth bill?

I reckon.

 

[Dj_Jestar]

probably to stop people associating ocuk with many of the discussions going on here.

 

[MikeHunt79]

Surely the bot wouldn't find much as you need to be signed up to view threads?

 

[BillytheImpaler]

Surely the bot wouldn't find much as you need to be signed up to view threads?

No you don't. You need to be logged in to post.

Google is kept out by the forum's robots.txt which explicitly forbids them from indexing the forum.

 

[NickK]

Problem is that there's numerous search engines out there that all periodically refresh their cache of your site so you get a considerable bandwidth bill from it.

 

[iCraig]

Forums are very dynamic. The data cached one day is very different the next. The Googlebot would keep coming back to index the new data, which is a largely pointless excerise seeing as the information in these threads is constantly expiring and being created. It's far easier to exclude the Googlebot from it. It's probably set as default in vBulletin's setup.

 

[toString]

There is no setting in vB.
But on my own forum I do notice google and yahoo bots are constantly looking at something. Thankfully it's only a small forum so I don't really care.

 

[lookitsjonno]

it'd probably effect ocuk's google ranking also with all of the outbound links in sigs etc

 

[lloydsj]

it'd probably effect ocuk's google ranking also with all of the outbound links in sigs etc

Negatively?