OVH - Fire destroys servers

[Gimpymoo]

https://www.reuters.com/article/us-france-ovh-fire-idUSKBN2B20NU

https://imgur.com/YPYC3JA

Surely systems should have been in place to prevent something like this happening in a modern day data centre?

 

[Deleted member 77746]

Surely systems should have been in place to prevent something like this happening in a modern day data centre?

Loads of solutions. I guess the suppression system either didn't go off or it was never installed.

 

[Quartz]

Surely systems should have been in place to prevent something like this happening in a modern day data centre?

It depends how and where the fire started.

But what concerns me is that this is a cloud data centre. If a client's instances go down, they should be restarted elsewhere in the provider's cloud. Downtime should be minimal; none of this disaster recovery malarkey.

 

[Hotwired]

One crispy data centre is a good way to demonstrate how resilient your service is.

 

[ChrisD.]

Apparently they are renowned for using custom servers, such as watercooled, open chassis etc.

 

[ChrisD.]

But what concerns me is that this is a cloud data centre. If a client's instances go down, they should be restarted elsewhere in the provider's cloud. Downtime should be minimal; none of this disaster recovery malarkey.

That’s not how it works and it will be baked into the SLA.

 

[Caged]

Cloud means you write code that can survive nodes disappearing

 

[Quartz]

That’s not how it works and it will be baked into the SLA.

Well yes you have to restore your VM and data to a new instance, but that instance can be anywhere.

 

[Rroff]

It depends how and where the fire started.

Indeed - if the source was external to and sustained from outside the server room environment internal suppression will only hold up so long.

I've done too many inspections where I've found things like fire doors propped open, etc. though - too many people are lazy, short sighted and stupid when it comes to health and safety especially when it is work stuff and not things they own themselves - even in facilities like this

 

[Semple]

Indeed - if the source was external to and sustained from outside the server room environment internal suppression will only hold up so long.

I've done too many inspections where I've found things like fire doors propped open, etc. though - too many people are lazy, short sighted and stupid when it comes to health and safety especially when it is work stuff and not things they own themselves - even in facilities like this

Suppression systems aren't always good either. I had a Dutch customer a couple of years ago who had their fire suppression system right next to their storage array.

Someone had set of a false alarm, and these suppression systems are insanely loud and cause immense vibrations. It damaged every disk in the array and resulted in several petabytes of data loss.

 

[Shac]

I cannot stop myself from saying ...

Intel inside.

Which considering you all seem to show Intel rigs may get me lynched!

 

[Hotwired]

Are they using shipping container offices?

I know construction workers do it but that's construction workers not a tech company.

 

[Cyber-Mav]

Those pics look like the building in first level of watchdogs 2.

 

[Yaayuh!]

We may have lost our secondary VPS...
Don’t keep all your eggs in one basket as the saying goes.

That’s not how it works and it will be baked into the SLA.

 

[Deleted member 77746]

We may have lost our secondary VPS...
Don’t keep all your eggs in one basket as the saying goes.

I hope you had backups.

 

[Yaayuh!]

I hope you had backups.

Of course, but irrelevant as it was only our secondary location. We’ll spin up another elsewhere later.

 

[ChrisD.]

Well yes you have to restore your VM and data to a new instance, but that instance can be anywhere.

That's up to the customer to sort and not the cloud provider, unless of course they pay the providor extra for it.

 

[spidiweb7]

Someones going to have a headache when the investigation concludes... my guess is that someone wasn't following procedure and propped a fire door open, it's usually something as simple as that.

 

[ChrisD.]

Someones going to have a headache when the investigation concludes... my guess is that someone wasn't following procedure and propped a fire door open, it's usually something as simple as that.

It was an unmanned DC, so I doubt it would be something like that.

As I said above OVH have been known to modify servers, watercool them, run them submerged, open case etc. I'd imagine either that, or an old battery somewhere swelling, exploding and inadequate fire suppression system combined with poor building design.

 

[Worthy]

Well yes you have to restore your VM and data to a new instance, but that instance can be anywhere.

So many people fall into this trap

That's up to the customer to sort and not the cloud provider, unless of course they pay the providor extra for it.

Yep, it's amazing marketing from MS/Amazon and co that makes people think HA between regions is just built in and provided for free.