Global BSOD

Associate
Joined
24 Mar 2020
Posts
208
Interesting how many companies are using CrowdStrike. They must have a fat contract with these government organizations.

Doubt many will be renewing now. Microsoft have got to be rubbing their hands ready to slap them all with another subscription.
They'll still be renewing it just means the bung to whoever gives the contract out has had a steep rise
 
Soldato
Joined
5 Mar 2003
Posts
10,768
Location
Nottingham
Fairly sure we don't just push down updates as soon as they are available. Often tested before a roll out. Unless something marked as security critical that needs pushed out asap due to a threat it seems bizarre this wasn't picked up in any testing. Especially when the actual issue is a complete BSOD of the system. Surely that gets picked up on the first batch of testing?
You're assuming it crashes on every single version of windows? I'd be shocked if thats the case. It might just be an older (more "stable") version of windows impacted. I can't imagine airlines and banks being on the latest and greatest version.

One thing that has shocked me is that they weren't on incremental rollouts. When I release new software for an environment that can't be thoroughly tested (i.e. android devices as there are so many different types) I'll do an incremental rollout of 20% per day and refresh the log table like I'm buying tickets for Glastonbury.
 
Associate
Joined
26 Jul 2007
Posts
940
IT support thousand yard stare after waking up to the Crowdstrike outage.


CokSu4J.jpeg
Yep that was me staring at hundreds of Servers BSOD at around 6:30am this morning.
 

Si.

Si.

Soldato
Joined
22 Oct 2002
Posts
2,652
Location
Melbourne, Aus
An anti virus company will be making 100s of releases a month - current products, previous products and the fact that the anti virus business is very dynamic considering its prey. The number of configurations for Windows software is insane - I have no doubts that this would have been thoroughly tested but you can't have 100% coverage for the millions (billions?!) of configurations for MS platforms - the hardware, the software, the different versions, the different drivers / applications on that machine. Will be really interesting to know what the actually problem was (if we'll ever be told) but bugs like this are constantly released into the wild. It's just this time it caused BSOD.
AV rarely works on definition files these days, updates are not as common as they used to be, It' all sandboxing now.
 

Si.

Si.

Soldato
Joined
22 Oct 2002
Posts
2,652
Location
Melbourne, Aus
You're assuming it crashes on every single version of windows? I'd be shocked if thats the case. It might just be an older (more "stable") version of windows impacted. I can't imagine airlines and banks being on the latest and greatest version.

This is impacting Windows 10 and 11 and several different feature release versions (Even older long term service editions use my banks for airlines like you mentioned). The problem is that Crowdstrike sits between the OS layers and intercepts executing code BEFORE the OS get's it. That's part of the appeal of how it works.

This is also why the BSOD is such a major impact, this literally destroys the OS by installing a faulty SYS file. The only way to rectify is to restore in recovery mode and manually remove the SYS file from the devices. Great if you are all in one office, not so great if you are a global country (Like where I work) with users all over the world and no local support team to implement this "Fix". If you are running server in UEFI boot mode (and a lot will be) then you can't get into recovery mode.. so you better hope your backups work.

Bye Bye Crowdstrike. This is a billion dollar mistake.
 
Soldato
Joined
14 Nov 2007
Posts
16,422
Location
In the Land of Grey and Pink
This is impacting Windows 10 and 11 and several different feature release versions (Even older long term service editions use my banks for airlines like you mentioned). The problem is that Crowdstrike sits between the OS layers and intercepts executing code BEFORE the OS get's it. That's part of the appeal of how it works.

This is also why the BSOD is such a major impact, this literally destroys the OS by installing a faulty SYS file. The only way to rectify is to restore in recovery mode and manually remove the SYS file from the devices. Great if you are all in one office, not so great if you are a global country (Like where I work) with users all over the world and no local support team to implement this "Fix". If you are running server in UEFI boot mode (and a lot will be) then you can't get into recovery mode.. so you better hope your backups work.

Bye Bye Crowdstrike. This is a billion dollar mistake.

Already cost them over 16 billion dollars in the shares value drop of over 20%.
 
  • Like
Reactions: Si.
Don
Joined
7 Aug 2003
Posts
44,411
Location
Aberdeenshire
The question is who approved it. Lone programmers don't release code in this size IT firm, this is a massive failure that would involve multiple people, from the people who coded it, the testers and the managers who approved the code commits and patch releases
It's seriously damaging to their company that this made it out into the wild. Their clients will be expecting an investigation and explanation as to how this happened and presumbly(!) how it bypassed all their procedural controls.

I know I triggered a few people earlier, but DEI is increasingly a bain to technical companies and is now on a shoogly peg as to whether it's even demonstrably good for companies performance, or indeed the people it's meant to serve.

Whether it was the case here or not remains to be seen, but many of us working in the tech sector will have stories about DEI and the weird and wonderful impact it has on rectruitment practises and promotion decisions.
 
Caporegime
Joined
22 Nov 2005
Posts
45,524
so can microsoft sue all the news and media channels who blatantly have head lines and videos with their name and microsoft logo?


microsofts reputation ruined due to a faulty buggy crowdstrike app that crashes pcs

Whether it was the case here or not remains to be seen, but many of us working in the tech sector will have stories about DEI and the weird and wonderful impact it has on rectruitment practises and promotion decisions.
Thats every company now, even trumps secret service agents etc..

there's women about 5ft tall who can't carry the president, they can't body block him etc... all they can do is standard there and look DEI
Hell they would struggle to bathe and change bidens pampers

The worlds pathetic
 
Last edited:
Caporegime
Joined
22 Nov 2005
Posts
45,524
Microsoft’s reputation won’t suffer all that much more from being associated with someone else’s buggy software.
most people won't get past the head line or ms logo, they are now associated and blamed for a failure they have nothing to do with and the news channels should be taught a lesson for jumping to conclusions.

it's different when they are telling us not to be arm chair experts, don't jump to conclusions , don't listen to the conspiracies that 50% of the time are true anyway.

They should get their facts straight before they rush out a headline.

we went from professional journalism with standards to it being playground gossip
 
Last edited:
Man of Honour
Joined
15 Jan 2006
Posts
32,425
Location
Tosche Station
It's seriously damaging to their company that this made it out into the wild. Their clients will be expecting an investigation and explanation as to how this happened and presumbly(!) how it bypassed all their procedural controls.

I know I triggered a few people earlier, but DEI is increasingly a bain to technical companies and is now on a shoogly peg as to whether it's even demonstrably good for companies performance, or indeed the people it's meant to serve.

Whether it was the case here or not remains to be seen, but many of us working in the tech sector will have stories about DEI and the weird and wonderful impact it has on rectruitment practises and promotion decisions.
In my recent conversations with recruitment in my company, they've said how it can be difficult when they're told "we need a female here"... :o and no before anyone asks this was ref: generic tech/admin jobs, not anywhere it might matter.
 
Soldato
Joined
3 Jun 2005
Posts
3,115
Location
The South
Thats every company now, even trumps secret service agents etc..

there's women about 5ft tall who can't carry the president, they can't body block him etc... all they can do is standard there and look DEI
Hell they would struggle to bathe and change bidens pampers

The worlds pathetic
Now look what you've done @Jokester, you've got @arknor started - we'll now have to endure listening to them tell us how bog roll is "woke" :rolleyes::cry:
 
Don
Joined
7 Aug 2003
Posts
44,411
Location
Aberdeenshire
In my recent conversations with recruitment in my company, they've said how it can be difficult when they're told "we need a female here"... :o and no before anyone asks this was ref: generic tech/admin jobs, not anywhere it might matter.
I was once at a place that recruiters almost blocked the recruitment of a physically disabled guy (who passed the competence test with flying colours) on the basis that they already had enough disabled people at the company. My pal was on the technical team and had to point out we don't recruit people because their disabled, but because they are competent.

50% female engineer recruitment was a thing for us for a year or two before that blew up in their face as well.

Anyway, back to Crowdstrike. Oopsy.
 
Caporegime
Joined
22 Nov 2005
Posts
45,524
how bog roll is "woke" :rolleyes::cry:
the bog roll isle doesn't look very diverse.
all white and of similar sizes.

can we go back to the 80s and 90s, "we are all the same on the inside" racism is only skin deep
it seemed a whole lot less racist back then.


now the message seems to come across like there is a difference and we aren't all the same after all
 
Last edited:
Soldato
Joined
22 Nov 2006
Posts
23,670
Should be a new bank holiday.

Interesting how many companies are using CrowdStrike. They must have a fat contract with these government organizations.

Doubt many will be renewing now. Microsoft have got to be rubbing their hands ready to slap them all with another subscription.

Given the way this software works and what it does, governments should not be using it. It would not comply with security requirements (not in the UK at least). Of course some will probably just do it anyway without checking..
 
Last edited:
Back
Top Bottom