Googlebot hammering site

chippie · 20 Nov 2006 at 18:30

Have just discovered that Googlebot is requesting the same page on our forum, and using up loads of our bandwidth.

The forum is small, only 3 or 4 normal visitors a day on average, but googlebot visited 18343 times yesterday. No, that's not a typo, eighteen thousand times.

I've tried blocking it in robots.txt, but it seems to be ignoring it.

What can I do to stop it?

chippie · 20 Nov 2006 at 18:41

robmiller said:
Are you positive it's Google? Googlebot respects robots.txt.

Yeah, it's definately google.

I put a robots.txt with a blanket disallow, but it seemed to ignore that.

Just changed it to a specific one just disallowing googlebot, and it now appears to have stopped.

chippie · 20 Nov 2006 at 18:50

Actually I was wrong. I also renamed the file it kept hammering, as well as changing robots. I have now put the file back and it has started up again. 18 times in the last minute.

Must have been hammering the 404 page for the past few minutes!

chippie · 20 Nov 2006 at 19:50

Contents of robots.txt is

User-agent: Googlebot
Disallow: /

Only been there less than 24 hours though, so googlebot probably hasn't got round to looking at it yet.

It is always the same page it looks at, which is part of the calendar system we have on the forum. It appears to be goint to the calendar_scheduler.php each time, but with a different session id each time, and geting itself all messed up.