WhatsApp Down?
- Thread starter Lysander
- Start date
for 6 hours, affecting Facebook's entire platform globally?
It's really ******* embarrassing. If anything - I think if it turns out to be malicious it's actually easier to understand - considering what's going on with Facebook right now... If it was just a scheduled change that went wrong - then that's a lot worse in my opinion, it would boil down to point blank incompetence.
I imagine we'll get 'something' whether it'll be the truth or not, who knows..
Maybe around the same time the whistleblowing/regulation story is about to hit the news again
They've posted this loose statement:
https://engineering.fb.com/2021/10/04/networking-traffic/outage/
https://engineering.fb.com/2021/10/04/networking-traffic/outage/
Associate
- Joined
- 14 Dec 2004
- Posts
- 1,205
It's amazing how so many people think this is another conspiracy, because everything is a conspiracy right?
Apparently no one screws up their job anymore
Apparently no one screws up their job anymore
Associate
- Joined
- 14 Dec 2004
- Posts
- 1,205
If you work in IT you must have come in contact with these people? I have, I'm normally the one who has to clear up their mess
They are normally at the lower end of the pay scale where a company would rather cheap out and not pay someone with experience
Soldato
We've all flashed a bad config to every appliance in the organisation at least once 
Facebook employs some of the best network engineers in the world, I know some of them personally - they absolutely do not cheap out in any way shape or form. Working in engineering at Facebook generally would be one of the most well paid jobs in the tech industry, especially if it involves automation (which it will).
This may well have been just some dumb change that some idiot pushed, who didn't get approval or somehow circumvented all the checks - who knows, I just think it is quite strange - when you look at not only the magnitude of the fault (entire global platform-down event) but also how they couldn't roll it back, that's the real worry in all of this - why didn't their automated post-checks abort it the moment it caused impact.
Whilst I'm generally not a believer in conspiracies, the fact the company is being dragged over the coals over this whole whistleblower thing can't really be overlooked, especially when it's a known fact that a large number of cyber attacks are down to disgruntled employees. And we know that Facebook has had it's share of whistleblowers, and upset employees over the years.
Associate
- Joined
- 14 Dec 2004
- Posts
- 1,205
Yeah, they intentionally took down their own platform, made international news, pushed lots of users away from their platform, had to post on a rival platform, damaged their shareholders pocket and all to bury some whistleblower a day AFTER it was released drawing MORE attention to it
Is that more believable than someone getting sloppy at their job?
Man of Honour
The fact that Facebook staff were locked out of their building because access cards system was also down is quite amusing lol.
I don't think that's what's being suggested, in the sense that Facebook intentionally did it do distract attention or whatever - that's just silly. However, a disgruntled employee with the right level of knowledge (a lone wolf) seems plausible to me, to cause additional disruption and to make a point - it's not exactly unheard of, and that person might have a motive. (poor performer, unfair treatment, some other reason,.... who knows) I can't prove it of course, but it doesn't seem that far fetched to me.
Of course - it might have been somebody being sloppy with their work or just a perfect storm which due to one tiny change caused a cascade failure, but that in my opinion is even worse - because companies like Facebook have highly sophisticated automation and enormous numbers of checks to catch out and auto-rollback stupid mistakes.
Companies like Facebook don't make network changes by sitting and typing things in manually, most if not all of their changes are spawned, modeled and simulated virtually before being pushed out. In theory anyway
Tbf I remember a senior manager at my old firm going bonkers when he was 4th or 5th in command to approve a change and it transpired he was the only one who opened the doc and it was some blokes dominos receipt he had accidently uploaded.
It had gone through 3 or 4 change approvers before it got to him.
It had gone through 3 or 4 change approvers before it got to him.
Associate
- Joined
- 14 Dec 2004
- Posts
- 1,205
I don't think that's what's being suggested, in the sense that Facebook intentionally did it do distract attention or whatever - that's just silly. However, a disgruntled employee with the right level of knowledge (a lone wolf) seems plausible to me, to cause additional disruption and to make a point - it's not exactly unheard of, and that person might have a motive. (poor performer, unfair treatment, some other reason,.... who knows) I can't prove it of course, but it doesn't seem that far fetched to me.
Of course - it might have been somebody being sloppy with their work or just a perfect storm which due to one tiny change caused a cascade failure, but that in my opinion is even worse - because companies like Facebook have highly sophisticated automation and enormous numbers of checks to catch out and auto-rollback stupid mistakes.
Companies like Facebook don't make network changes by sitting and typing things in manually, most if not all of their changes are spawned, modeled and simulated virtually before being pushed out. In theory anyway
Disgruntled employee is plausible but they'd have to be pretty stupid, that sort of action gets you immediately fired and the seriousness of the action would also get you arrested. That persons career and life would be over and yes they would get found out, because there would be a audit trail.
Yea, what if tomorrow President Putin "screws" his job up and presses that button for the nuclear war heads.
How would you feel?
Associate
- Joined
- 14 Dec 2004
- Posts
- 1,205
Hot (for a microsecond)
Contingency Plans: Angle Grinder (in case of loss of card ID access to rack)I've submitted a change request at work this week that involves our BGP routing. I suspect I will be grilled on it more then usual.
Commissario
If that is true it probably says a lot about how over confident FB were that their network would never, ever go do in such a way that it would affect their access system (or how they never thought of the access system itself failing).
Which also reminds me of the way that some people always assume new, higher technology solutions to simple issues are best and how they assume there will not be a time when that technology fails for any reason.
You'd think with any access system there would still be at least one physical way to gain entry even if the computer fails, be it a one time code on a pad that bypasses the central system or some physical key (both of which could be set to sound a physical alarm if used), and potentially as important, one person either on site or within a short distance who can use that access regardless of what time/day it is.
High levels of automation are great, until you make it so that the tools needed to fix issues are also automated and reliant on the system that has fallen over.
I had visions last night of some grizzled old network admin that everyone laughed at going through an ancient and worn rolodex to find the only physical copies of key staff's contact details and calling them (on a rotary phone
).Yes, you are right. Look at the Tesla cars, for example, and how they get software fixes over the network for hardware problems.
Not to mention that the Model 3 hasn't actually gotten that classic speedometer board, but instead relies on a vertical display which is not comfortable when driving, because your eyes should move too much to the right.
It isn't safe, it's actually quite a bit dangerous when driving.
Not to mention that the Model 3 hasn't actually gotten that classic speedometer board, but instead relies on a vertical display which is not comfortable when driving, because your eyes should move too much to the right.
It isn't safe, it's actually quite a bit dangerous when driving.