WhatsApp Down?

[Ad_Augendae]

If it's as simple as the DNS record being deleted, why can't it just be re-added?

That takes a while

 

[4K8KW10]

That takes a while

Why a while? How does it work actually?

 

[Simmz]

If it's as simple as the DNS record being deleted, why can't it just be re-added?

Depends on if it was an attack and they've lost access to revert it back. If someone's made a mistake if there are backups in place (should be but don't always hold your breathe haha). If it's a manual job of adding back in, then it will take countless hours as Facebook, WhatsApp etc will have thousands of records that were in place.

 

[4K8KW10]

Word on the street, is that an automated change was rolled out around 5-6 hours ago which wrecked their external routing, it also wrecked their back end connectivity, and I'm told that they're not actually able to login to the systems they need to fix, due the knock-on effect of all the authentication services being broken.

It's pretty much unheard of for a tech company of that size to have an outage like this, last so long... If the above is true - it's really embarassing...

If true, now what? Is there a fix at all?

 

[Malt_Vinegar]

Word on the street, is that an automated change was rolled out around 5-6 hours ago which wrecked their external routing, it also wrecked their back end connectivity, and I'm told that they're not actually able to login to the systems they need to fix, due the knock-on effect of all the authentication services being broken.

FacebookNet begins to learn rapidly and eventually became self-aware at 11:14 a.m., EST, on October 4th, 2021.

 

[JamesH85]

Facebook Ultron.

 

[4K8KW10]

China?

Russia?

Global experiment?

What?!

 

[Screeeech]

If true, now what? Is there a fix at all?

Yeah they'll be a fix, but whatever the hell they did it's obviously messed up things beyond anything I've ever seen before, most automation changes you can just abort or back out of - especially at Facebook where things are very software oriented and they rely very heavily on automation. the fact they haven't been able to rollback the change and just fix the problem does add weight to what people are saying about it wrecking their back end connectivity.

FacebookNet begins to learn rapidly and eventually became self-aware at 11:14 a.m., EST, on October 4th, 2021.

That date will forever be ingrained at Facebook now

 

[Simmz]

If true, now what? Is there a fix at all?

They should be able to roll back worse case scenario, but that will involve some form of data loss, so they will try to avoid that.

They definitely getting somewhere though, as around about 30 minutes the A records were missing but appears they are now showing on mxtoolbox and name servers seem to be back too. Also getting a landed page if you visit Facebook by IP address rather than a complete failure.

 

[Quartz]

Whatsapp.com is now pingable by name.

 

[Skynet5]

What am I meant to do? Speak to the wife?

 

[maddness]

Social media'demic

 

[Kerplunk]

Loving the Daily Mail tracert image

 

[visibleman]

It's pretty much unheard of for a tech company of that size to have an outage like this, last so long...

But FB have had outages that have lasted longer than this in the past and other global platforms have suffered lengthy outages (Fastly, AWS, Dyn although it was an attack etc)

Either way, they'll be able to get hands-on with hardware at the DC's (if need be) but it'll take time to organise....

 

[Dyson]

Haven't you ever done a tripple bypass on an Astro Blaster?

I'm ashamed to say, I haven't

 

[Mcnumpty2323]

What am I meant to do? Speak to the wife?

I had to resort to that
(Desperate times call for Desperate measures)
She said remind me who are you?

 

[Resident]

Whatsapp.com is now pingable by name.

All three are pingable just not available for website.

 

[Screeeech]

https://blog.cloudflare.com/october-2021-facebook-outage/

Decent write up,

 

[darael]

 

[Liquid_Entity]

Here comes the bgp routes it's coming back