Global BSOD

[Mercurial]

And the ******* thing about all this is that the advent of automation means smaller and smaller IT teams are looking after bigger and bigger estates.... then suddenly along comes something where the fix cannot be automated ....

In my role i've seen it happen, since roughly around the end of the 00s, companies have downsized or deskilled their own IT departments (partially I imagine because of cost, but also the increased complication and specialisation of running all the IT functions). This work has been farmed out to 3rd parties, good luck with that on a widespread incident such as today when I imagine everyone's ticket is P1 and there's only a finite amount of bodies they can throw at a problem (hence why it's cheaper in the first place . Oh, the irony.

 

[Pawnless Endgame]

Today is the day most schools in England break up. I know it's a worldwide issue but I can't help thinking this is deliberate to tank the tourism industry, even if the press isn't making out that it's malicious.

 

[SupraWez]

And the ******* thing about all this is that the advent of automation means smaller and smaller IT teams are looking after bigger and bigger estates.... then suddenly along comes something where the fix cannot be automated ....

This ^

For a lot of fixes this will require someone in front the box, if there are support staff on site they could have 1000s of machines no longer reachable remotely

 

[john_smith]

Today is the day most schools in England break up. I know it's a worldwide issue but I can't help thinking this is deliberate to tank the tourism industry, even if the press isn't making out that it's malicious.

You really believe that?

 

[visibleman]

Today is the day most schools in England break up. I know it's a worldwide issue but I can't help thinking this is deliberate to tank the tourism industry, even if the press isn't making out that it's malicious.

 

[dlockers]

Today is the day most schools in England break up. I know it's a worldwide issue but I can't help thinking this is deliberate to tank the tourism industry, even if the press isn't making out that it's malicious.

 

[Dirk Diggler]

You should be straight onto the OS drive, if not try a few drive letters till you find the directory.

Depending on the build of the machine, if its been upgraded etc, you can try the following

c:\windows\system32\drivers\crowdstrike
d:\windows\system32\drivers\crowdstrike

even had one machine where it was on x:, no idea how that was built :-

x:\windows\system32\drivers\crowdstrike

Then once in there you can delete the file causing the issue :-

dir C-00000291*.sys
del C-00000291*.sys

It's not showing any volumes in DISKPART, I suspect because the drive is encrypted.

 

[MCFC_ANDY]

gora love virgin broadband today, yes your internet is down, well our system is down so cant check what is the issue thankfully my BT is still active and working before contract ends

 

[grudas]

Today is the day most schools in England break up. I know it's a worldwide issue but I can't help thinking this is deliberate to tank the tourism industry, even if the press isn't making out that it's malicious.

it's not that deep.. rubbish gets pushed to production all the time. The difference is this one causes a BSOD, that's it.

 

[S.A.S]

I saw a headline earlier saying 'Microsoft has a lot to answer for'. With relatively little experience of these things, is this true? As it was a third party companies update? I've had a friend say yes due to certain 3 letter requirements and regulations.

 

[SupraWez]

It's not showing any volumes in DISKPART, I suspect because the drive is encrypted.

Could be, if using bitlocker or something there will be different instructions.

 

[radderfire]

https://www.crowdstrike.com/blog/statement-on-windows-sensor-update/

Link corrected above.

 

[squerble]

I saw a headline earlier saying 'Microsoft has a lot to answer for'. With relatively little experience of these things, is this true? As it was a third party companies update? I've had a friend say yes due to certain 3 letter requirements and regulations.

Awful media gonna awful media.

Microsoft have nothing to answer for regarding the BSOD issue. They had their own Azure outage a few hours earlier but that seems to be unrelated.

 

[mid_gen]

I saw a headline earlier saying 'Microsoft has a lot to answer for'. With relatively little experience of these things, is this true? As it was a third party companies update? I've had a friend say yes due to certain 3 letter requirements and regulations.

Nothing Microsoft can do if people install third-party software that ****s all over people's systems.

Super easy way to see which companies have competent IT departments that roll out to staging environments and test anything like this before it hits prod.

 

[Grim5]

gora love virgin broadband today, yes your internet is down, well our system is down so cant check what is the issue thankfully my BT is still active and working before contract ends

Crowd strike is a security software right and it's brilliant really, because you can't be hacked if your PC isn't on, so really the update is just working at 200% efficiency

 

[LizardKing]

Tbf MS also have a history of releasing updates that have caused havoc, not long back a defender update kindly deleted a ton of shortcuts that needed to be recovered from the shadow copies. (Not this level granted)

But this is amazing incompetence of the highest order, CS must do zero testing of their definition updates for this to get out so quickly.

 

[MCFC_ANDY]

Crowd strike is a security software right and it's brilliant really, because you can't be hacked if your PC isn't on, so really the update is just working at 200% efficiency

even better i can play my game with internet down and not get hacked, its a win in my book

 

[visibleman]

It's not showing any volumes in DISKPART, I suspect because the drive is encrypted.

If this is a local system and it is BitLocker'd, then access CMD through recovery mode where you should be asked the BitLocker key.
Alternatively, you may be able to force safe mode using bcdedit or constantly rebooting the system, hoping the network stays up long enough for the agent to download the new update.

Saying that, i would look on CrowdStrike's support portal as it looks like, from some screenshots floating around, they have a support page dedicated to the incident with instructions for a fix including BitLocker'd systems.
Alternatively contact your CrowdStrike rep and have a moan.

 

[SupraWez]

or constantly rebooting the system, hoping the network stays up long enough for the agent to download the new update.

It was mentioned on radio earlier that some techs were advising up to 15 reboots to get the update

 

[visibleman]

"Oh dear, oh dear"