Global BSOD

[Raiden85]

Antivirus stuff usually has access to all.

Can't remember the brand now, but I had a 3rd antivirus update many years that caused Windows to go into a boot loop after an update to it, had to enter safe mode and remove it in order to get Windows to boot normally. Quite a few things can cause this but they really shouldn't be allowed to if its possible. It is a bit worrying that if my Avast AV did a really bad update then my OS could be bricked on reboot that even safe mode may not easily get me out of. If MS has to significantly lock down parts of the OS in order to prevent any old 3rd party programs breaking the OS this easily then it really needs to be done.

 

[CGrieves]

"hello could we interest you in a crowdstrike subscription?"

That's the real shame of it all. This utterly stupid incident will (rightly) destroy any confidence in what I feel is one of the best endpoint security vendors. We've had more decent actionable threat intelligence from CS than any other tool I've used in my career.

 

[ED209]

By the end of the day most of our systems where back up and running, hopefully by Monday all will be right again

 

[coupe69]

hopefully by Monday all will be right again

Really unlikely though isn't it, the surgery where my wife works will have to wait for their IT guys to come out and physically log onto machines, sort encryption, and delete the rogue file. Multiply this by many thousands of doctors' surgeries nationwide and it could be some time before they're all back up and running.

 

[Jimbeam3678]

Still at Schiphol airport trying to get back from Turin.

 

[Grim5]

Really unlikely though isn't it, the surgery where my wife works will have to wait for their IT guys to come out and physically log onto machines, sort encryption, and delete the rogue file. Multiply this by many thousands of doctors' surgeries nationwide and it could be some time before they're all back up and running.

Yup, services are coming back online but there will still be millions of PCs that need to be manually rebooted and potentially have the bad file removed and encryption makes it take even longer and a human has to do this one by one for each pc - could be a week or two for every pc in the world affected to be fixed

 

[CGrieves]

Can't remember the brand now, but I had a 3rd antivirus update many years that caused Windows to go into a boot loop after an update to it, had to enter safe mode and remove it in order to get Windows to boot normally. Quite a few things can cause this but they really shouldn't be allowed to if its possible. It is a bit worrying that if my Avast AV did a really bad update then my OS could be bricked on reboot that even safe mode may not easily get me out of. If MS has to significantly lock down parts of the OS in order to prevent any old 3rd party programs breaking the OS this easily then it really needs to be done.

You have to consider that any threat actor exploiting a buffer overflow or RCE (which in the vast majority of cases is not Microsoft's fault) will be operating at the lowest level possible, ideally at the kernel level. Our security software needs to operate at the same level, otherwise it'll be useless.

 

[dlockers]

That's the real shame of it all. This utterly stupid incident will (rightly) destroy any confidence in what I feel is one of the best endpoint security vendors. We've had more decent actionable threat intelligence from CS than any other tool I've used in my career.

Tbf PCs are totally impenetrable now.*

*i.e. they can't even boot up

 

[chrismscotland]

Still at Schiphol airport trying to get back from Turin.

Wow that's been some trek today!

 

[Mercurial]

Can't remember the brand now, but I had a 3rd antivirus update many years that caused Windows to go into a boot loop after an update to it, had to enter safe mode and remove it in order to get Windows to boot normally. Quite a few things can cause this but they really shouldn't be allowed to if its possible. It is a bit worrying that if my Avast AV did a really bad update then my OS could be bricked on reboot that even safe mode may not easily get me out of. If MS has to significantly lock down parts of the OS in order to prevent any old 3rd party programs breaking the OS this easily then it really needs to be done.

It was Kaspersky. It was also when I stopped using AV on my own personal machines.

I used to do server disaster recoveries and one of the biggest problems with bringing a machine back from backup was if an AV was installed (especially Norton Endpoint & McAfee being the main culprits, Sophos/Nod which were common then were no problem at all).

 

[Jimbeam3678]

Wow that's been some trek today!

Managed to grab the last flight back. Sat on the runway now. KLM staff have been great to be fair to them. Some ugly scenes earlier where the local police had to get involved, they weren’t ******* about

 

[ED209]

Really unlikely though isn't it, the surgery where my wife works will have to wait for their IT guys to come out and physically log onto machines, sort encryption, and delete the rogue file. Multiply this by many thousands of doctors' surgeries nationwide and it could be some time before they're all back up and running.

Our IT sorted out the services which had failed, all staff laptops where working fine just could not access certain sites\services that we use

 

[chrismscotland]

Managed to grab the last flight back. Sat on the runway now. KLM staff have been great to be fair to them. Some ugly scenes earlier where the local police had to get involved, they weren’t ******* about

Good luck getting back!

 

[explicit4u]

As expected, despite being hit heavy, most staff just enjoyed the Friday sun instead of complaining about IT issues. SQL servers, domain controllers, VPN, our warehouse system, and M365 all down at some point. Think this might force us out of CloudStrike which has been a long time coming.

Must have been a lucrative day for contractors.

 

[ED209]

As expected, despite being hit heavy, most staff just enjoyed the Friday sun instead of complaining about IT issues. SQL servers, domain controllers, VPN, our warehouse system, and M365 all down at some point. Think this might force us out of CloudStrike which has been a long time coming.

Yeah, Our VPN went down (UK 1) but our EMEA VPN still worked so we switched over to that. Unsure if we will be sticking with them or moving on.

 

[Jimbeam3678]

I would imagine most companies will stay with them once the post mortem has been completed and the required palms have been greased.

 

[Dirk Diggler]

I've managed to fix my machine but the majority of the office is still borked and unlikely to be fixed without an actual IT person travelling out to do it in person, which makes this the perfect outage, because a fix that can't be deployed remotely.

 

[ubersonic]

The backlog at some airports could get horrendous, with kids in England and Wales having presumably broken up for hols today.

Not to mention NHS and GP issues.

Correction.

NHS England and GP issues

 

[Mr Jack]

What on an end user desktop uses CrowdStrike?

It was CrowdStrike "falcon", which the server automatically deploys to end user PCs in order to provide protection across the entire network, that they stuffed up.

 

[Rroff]

My mum went to the doctors and their systems were mostly down and having to do it with pen and paper and having to call people manually rather than use the electronic board - sounded like they wouldn't be back to normal until after the weekend. Friend who is travelling in Europe at the moment had to find a hotel because their trains were cancelled and/or massively delayed and again didn't sound like things would be back to normal until Monday.