Yelp, thanks Microsoft - a day of MS08-067 hell

Stolly · 27 Oct 2008 at 19:38

This one is been taken very seriously, the place has been in a state of near panic since Friday and over the weekend.

This must be costing thousands at the place i am at, whole teams stopping what they were doing and moving onto getting this patch on. Its not the installation thats the problem, its the reboot thats the pain. Some boxes just don't have outage windows....

Anyone else in a patch frenzy ?

morfmedia · 27 Oct 2008 at 21:51

Not a frenzy per se but I think it'll be deployed pretty quickly in most places. As soon as MS say there is a wormable patch everyone quite rightly gets it out ASAP, especially as they've already seen targeted attacks.

Hopefully you're getting some overtime ;-)

bigredshark · 27 Oct 2008 at 21:54

No frenzy, deployment was to outside facing boxes only straight off. About 40 odd servers immediately, they all have failover boxes though so no impact at all on live services.

sidethink · 28 Oct 2008 at 09:09

Yep - no real drama here. Updated all the edge boxes on Friday night and all our workstations and laptops (virtually all XP) on Monday morning.

The other servers are being done through the course of this week.

mdjmcnally · 28 Oct 2008 at 13:59

Not a problem for me. Just patched the Check Point firewall with SMARTDefense to sort it.

Then can roll out the MS patch normally.

Sarduk · 28 Oct 2008 at 14:00

Thanks Microsoft for fixing a serious security flaw so quickly.

Also, if it is costing your organisation thousands, you have a lousy IT infrastructure and you suck.

Burnsy2023 · 28 Oct 2008 at 14:59

JimpsEd said:
Thanks Microsoft for fixing a serious security flaw so quickly.

Also, if it is costing your organisation thousands, you have a lousy IT infrastructure and you suck.

Also, it costs a lot more by MS not patching it so quickly and you getting your security comprimised.

morfmedia · 28 Oct 2008 at 17:31

JimpsEd said:
Thanks Microsoft for fixing a serious security flaw so quickly.

Also, if it is costing your organisation thousands, you have a lousy IT infrastructure and you suck.

So in a muti hundred server / several thousand PC environment the patches get tested and rolled out for free with no drama?

Good luck in dream world

Sarduk · 28 Oct 2008 at 17:34

morfmedia said:
So in a muti hundred server / several thousand PC environment the patches get tested and rolled out for free with no drama?

Good luck in dream world

Wow, a multi hundered?? That must mean, hundreds!

PC environment the patches get tested and rolled out for free with no drama?

Yeah, that is the idea. It happens at least once a month. Test, publish, automatic rollout (using McAffee's administration suite). Not really difficult.

GarethDW · 28 Oct 2008 at 17:41

JimpsEd said:
Thanks Microsoft for fixing a serious security flaw so quickly.

Also, if it is costing your organisation thousands, you have a lousy IT infrastructure and you suck.

Yay for the sweeping generalisations :rolleyes:

I work in an environment where downtime isn't allowed. We have many thousands of servers that require patching. As someone with such insightful knowledge as you surely knows, patching that number of servers requires a scheduled deployment strategy. Stopping from playing Solitaire for 5 mins whilst you patch a handful of servers isn't comparable.

Sarduk · 28 Oct 2008 at 17:44

GarethDW said:
Yay for the sweeping generalisations

I work in an environment where downtime isn't allowed. We have many thousands of servers that require patching. As someone with such insightful knowledge as you surely knows, patching that number of servers requires a scheduled deployment strategy. Stopping from playing Solitaire for 5 mins whilst you patch a handful of servers isn't comparable.

Well, delay it then? Also, if you're running the latest version of Windows Server (2008), MS08-067 is only an important and not a critical patch.

morfmedia · 28 Oct 2008 at 17:55

I think you either don't work in Enterprise IT or have a very free reign.

I doubt you'll find many corporates with serious server 2008 deployments yet

/Edit: I'm pretty sure it took Microsoft a few weeks at least from initially discovering the exploit, possibly in the wild to delivering patches.....

bigredshark · 28 Oct 2008 at 19:30

GarethDW said:
Yay for the sweeping generalisations

I work in an environment where downtime isn't allowed. We have many thousands of servers that require patching. As someone with such insightful knowledge as you surely knows, patching that number of servers requires a scheduled deployment strategy. Stopping from playing Solitaire for 5 mins whilst you patch a handful of servers isn't comparable.

Yeah and we deployed to every public facing server for a FTSE100 company (I'm less concerned about internal servers for now) with zero interruption to service, if you have that many servers and don't have clustering, NLB or hardware load balancing you need to fire the person who designed that as a priority.

Stolly · 28 Oct 2008 at 21:04

JimpsEd said:
Thanks Microsoft for fixing a serious security flaw so quickly.

Also, if it is costing your organisation thousands, you have a lousy IT infrastructure and you suck.

lol, well when you have 20k+ wintel servers and 100k+ desktops the people rolling out the out of band patch will want paying, hence the cost.

Not all can be auto-deployed due to the nature of what they do, and there will a percentage casualty rate, and its the reboot thats the problem.

Stolly · 28 Oct 2008 at 21:05

bigredshark said:
Yeah and we deployed to every public facing server for a FTSE100 company (I'm less concerned about internal servers for now) with zero interruption to service, if you have that many servers and don't have clustering, NLB or hardware load balancing you need to fire the person who designed that as a priority.

Hundreds of clusters, but not every server can be clustered because ultimately the customers won't pay for it.

bigredshark · 28 Oct 2008 at 21:17

Stolly said:
Hundreds of clusters, but not every server can be clustered because ultimately the customers won't pay for it.

Then they accept downtime as a result when work has to be done surely. You can't say 'no downtime' and then refuse to pay for resiliency.

Stolly · 28 Oct 2008 at 21:22

bigredshark said:
Then they accept downtime as a result when work has to be done surely. You can't say 'no downtime' and then refuse to pay for resiliency.

Customers say the craziest things don't they ?

Deathwish · 28 Oct 2008 at 23:24

Deployed to all servers as of tonight and desktop machines as soon as SCCM gets it done.

Job Done

GarethDW · 29 Oct 2008 at 00:33

Stolly said:
lol, well when you have 20k+ wintel servers and 100k+ desktops the people rolling out the out of band patch will want paying, hence the cost.

Not all can be auto-deployed due to the nature of what they do, and there will a percentage casualty rate, and its the reboot thats the problem.

Stolly said:
Hundreds of clusters, but not every server can be clustered because ultimately the customers won't pay for it.

+1 ...although our environment isn't our customers'. However, without going into detail, not all of it can be clustered for various reasons.

Like you say, it's the reboot that's the problem.

LordSplodge · 29 Oct 2008 at 08:58

JimpsEd said:
Also, if it is costing your organisation thousands, you have a lousy IT infrastructure and you suck.

Did somebody switch off your tack gene?

It could cost thousands in overtime. Saying somebody sucks and they have a lousy infrastructure based on one post beggars belief.

Yes in an ideal world it will be business as usual with regards to patch deployment but as often is the case the real world is nothing like this as management, office poltics and customers get in the way to say the least.