Any webhost type people here? Getting spammed by Facebook bots!?

NeilF

NeilF

NeilF

Soldato
Joined
15 Nov 2003
Posts
14,400
Location
Marlow
I noticed an area of my website constantly getting hit from "facebookexternalhit/1.1" and "meta-externalagent/1.1". Tens of hits a minute (at least), all the time...

I've added in entries to ROBOTS.TXT but reading that facebook often ignores those, I even altered my .HTACCESS file to ban them fullstop, which seemingly has worked.

Anyone else suffering from this? Here's a random post I found via google complaining of the same thing - https://news.ycombinator.com/item?id=41090130
 
Yes, sadly, I've recently reimplemented custom ModSecurity rules to rate-limit (and block, if necessary) across all services I host for customers.
It's been around for a good few years ago but has come to light again in the past couple of months. It is very easy to use facebook + googlebot as a DoS tool that can fly under the radar for usual anti-bot/DoS protection unfortunately.

I can see 260,000 hits in the past 24 hours across all of my own servers that I look after just from facebookexternalhit alone.

Blocking in .htaccess does the job perfectly fine for single sites.
 
Last edited:
Sort of nice to know it wasn't just me then...

"facebookexternalhit/1.1" were definately the huge majority, but in there were "meta-externalagent/1.1" ones too. So I just .htaccessed them both across my site...
 
Robots.txt is voluntary compliance meaning the bot doesn’t need to adhere to it.

You’ll need to update nginx or better yet, do as Beansprout said and sign up to Cloudflare. You can add the bots to the firewall rules easily.
 
You must log in or register to reply here.
Back
Top Bottom