WhatsApp Down?

Soldato
Joined
30 Nov 2003
Posts
2,992
If it's as simple as the DNS record being deleted, why can't it just be re-added?

Depends on if it was an attack and they've lost access to revert it back. If someone's made a mistake if there are backups in place (should be but don't always hold your breathe haha). If it's a manual job of adding back in, then it will take countless hours as Facebook, WhatsApp etc will have thousands of records that were in place.
 
Permabanned
Joined
2 Sep 2017
Posts
10,490
Word on the street, is that an automated change was rolled out around 5-6 hours ago which wrecked their external routing, it also wrecked their back end connectivity, and I'm told that they're not actually able to login to the systems they need to fix, due the knock-on effect of all the authentication services being broken.

It's pretty much unheard of for a tech company of that size to have an outage like this, last so long... If the above is true - it's really embarassing...

If true, now what? Is there a fix at all?
 
Joined
20 Oct 2002
Posts
16,167
Location
In a house
Word on the street, is that an automated change was rolled out around 5-6 hours ago which wrecked their external routing, it also wrecked their back end connectivity, and I'm told that they're not actually able to login to the systems they need to fix, due the knock-on effect of all the authentication services being broken.

FacebookNet begins to learn rapidly and eventually became self-aware at 11:14 a.m., EST, on October 4th, 2021.
 
Soldato
Joined
29 Dec 2014
Posts
4,603
Location
16509
If true, now what? Is there a fix at all?

Yeah they'll be a fix, but whatever the hell they did it's obviously messed up things beyond anything I've ever seen before, most automation changes you can just abort or back out of - especially at Facebook where things are very software oriented and they rely very heavily on automation. the fact they haven't been able to rollback the change and just fix the problem does add weight to what people are saying about it wrecking their back end connectivity.

FacebookNet begins to learn rapidly and eventually became self-aware at 11:14 a.m., EST, on October 4th, 2021.

That date will forever be ingrained at Facebook now :D
 
Soldato
Joined
30 Nov 2003
Posts
2,992
If true, now what? Is there a fix at all?

They should be able to roll back worse case scenario, but that will involve some form of data loss, so they will try to avoid that.

They definitely getting somewhere though, as around about 30 minutes the A records were missing but appears they are now showing on mxtoolbox and name servers seem to be back too. Also getting a landed page if you visit Facebook by IP address rather than a complete failure.
 
Associate
Joined
3 Jun 2005
Posts
2,067
Location
The South
It's pretty much unheard of for a tech company of that size to have an outage like this, last so long...

But FB have had outages that have lasted longer than this in the past and other global platforms have suffered lengthy outages (Fastly, AWS, Dyn although it was an attack etc) :confused:

Either way, they'll be able to get hands-on with hardware at the DC's (if need be) but it'll take time to organise....
 
Soldato
Joined
10 Jul 2010
Posts
5,123
48755423-10058445-image-a-72-1633372644137.jpg


48755425-10058445-image-a-76-1633372676181.jpg
 
Top