Global BSOD

[CaNsA]

Correction.

NHS England and GP issues

Correction.

Since then, various NHS Trusts across the UK have publicized their use of Varian kit. These include Barking, Havering, and Redbridge University Hospitals, East and North Hertfordshire NHS Trust, and the Beatson West of Scotland Cancer Centre (part of the NHS).

Source: https://www.theregister.com/2024/07/19/crowdstrike_update_nhs_it_outages/

 

[Mr Jack]

The fact that a software update not from MS can cripple the entire OS from even booting is just really crappy design. I don't care whether this is a server OS or a consumer one it really shouldn't happen. MS needs to make the OS far more resilient to this in the future.

Drivers for every operating system can do this. MS gives greater liberty to services like CrowdStrike - and that should be looked at - but every OS has vulnerabilities.

 

[Rroff]

One of the things I liked about RISC OS - could always boot unstrapped from the ROM if the software environment got stuffed up in some way. Albeit that is more complicated when you have a larger range of possible base hardware configurations and require 3rd party drivers, etc.

What I do think this demonstrates is how pants on head stupid MS's (and other parties) approach to updates and automatic updates, especially forced, is, even in corporate environments which roll out stable tested images on more critical systems.

 

[wildman]

Question is why? Why this specific update? Or is it "common" practise to push updates bypassing QA testing? Again, why? Either way, a colossal boo boo.
I've spoken to ex-colleagues today and it has, as expected, been an utter nightmare alongside bitlocker'd end-points. Never been so glad to be retired!

e: gibberish on first line.

its only common practice for companies running waffer thin IT departments. I work in Pharma and my compnay was totally unaffected because we don't just blindly accept all security updaters. All updates get rinsed in snadbox first using clones of our production encironments ebfore anyone even thinks about signing off approvals to push to production. the reaosn its such a big problem is becuase of a combination of greed and laziness. A lot of big corporations don't bother even sandboxing windows updates and jsut let them get auto pushed to live environments for the sole reason that its cheap and easy. Why pay for someone to look at testing something that comes from a big multinational conglomorate ?????

 

[wildman]

Still at Schiphol airport trying to get back from Turin.

get a cab to Centraal and go the the grasshopper and blaze up a fatty ? Purple sensemelia is good this time of year has to be afghani blend though

 

[LizardKing]

its only common practice for companies running waffer thin IT departments. I work in Pharma and my compnay was totally unaffected because we don't just blindly accept all security updaters. All updates get rinsed in snadbox first using clones of our production encironments ebfore anyone even thinks about signing off approvals to push to production. the reaosn its such a big problem is becuase of a combination of greed and laziness. A lot of big corporations don't bother even sandboxing windows updates and jsut let them get auto pushed to live environments for the sole reason that its cheap and easy. Why pay for someone to look at testing something that comes from a big multinational conglomorate ?????

Im intrigued, your essentially saying your company tests every av definition update? bearing in mind these get released at least once a day with more on occasion? not many company's have that sort of ability (they may well invest in more now after this) and your very much in the minority if you are testing such updates.

One thing for sure is a lot of company's BCP's are going to be getting a refresh! we were not even hit but we will be adding a recovery plan for such an incident going forward. it might just involve shipping a usb stick with a windows installation on and letting autopilot do the rest but this incident has opened the eyes of a lot of organisations no doubt.

 

[ubersonic]

Correction.

It doesn't say which specific hospital had the issue just "A UK hospital" however I would bet money it's in England

Also, one hospital is a lot more than I would have expected to have issues with their Varian kit over this considering Varian's upgrade cycle speed (they're currently still working on transitioning their kit from Windows 7 to 10, no joke).

 

[Blackjack Davy]

According to one source, known to me, Crowdstrike bypassed their testing and pushed it directly without going through their normal process.

I bet someone somewhere is in very hot water indeed.

 

[dowie]

Drivers for every operating system can do this. MS gives greater liberty to services like CrowdStrike - and that should be looked at - but every OS has vulnerabilities.

While this is true and indeed Mac has had some issues with previous OS releases and stuff like Adobe software has caused some crashes before it doesn't have the BSOD thing Windows has.

If a MacBook crashes and someone is trying to access a GP appointment system or whatever then just restart it. More rarely they have kernel panic but again... reboot. The BSOD thing here required people to know to reboot in safe mode, delete a system file etc.. and often they don't have access to that or need to faff around and wait for IT end result GP receptionists, train people etc.. all basically locked out of their windows machines until IT sorts it.

 

[Oh Ed not again.]

Fail fast, fix fast has a lot to answer for.

That or proper PR aren't reviews aren't happening. I'd wager both.

 

[dowie]

It was some file full of zeroes, some basic automated testing should have caught this:

https://twitter.com/i/web/status/1814299095261147448