TSO Down?

[Izi]

anyone else?

 

[Izi]

Apparently network issues, I thought they had redundancy for this.

 

[Izi]

Sounds bad:

https://twitter.com/x/status/48330257689874432

nervous right now!

 

[Izi]

backup now?

 

[Izi]

I've got email back too but the websites are still down.

my VDS's are down.

I assume their network will need some manual help being restarted.

With over 11k sites on the cluster, I can image the whole thing being down won't be fun.

 

[Izi]

Twitter, so it could be cr*p. Apparently it was a UPS failure causing the fire alarms to activate, in turn tripping the power and the DC is now running on generators until the UPS's can be checked over.

Edit - Just read it over on EtherClear blog; http://www.etherclear.co.uk/blog/company/service-status/bluesquare-23-outage/

Edit 2 - Twitter Hash - #Bluesquare

Hmm, running of generator is never good is it? I hope there is enough power.

 

[Izi]

The datacenter should be able to run on the generator for a long time as long as they have a fuel supply.

It is the switch over back to the UPS you want to worry about.

Hopefully they will do that during the night... At least that way we can be peacefully unaware (except the 100 SMS I will get from pingdom - nothing like a wakeup call than a "XXX.com is down".)


I don't know about blue square, but most say they carry fuel for 2 weeks? Is this BS?

 

[Izi]

I did see a Tweet that bluesquare is the Lidl of Data centers. I want to know what they mean by that.

 

[Izi]

Majority of my servers are back up; hopefully that'll be the last of the outages.

Fingers crossed, but be cautious - currently on generator power.

 

[Izi]

Anyone having issues with ftp ? I can currently connect but I get 0 files and folders listed.

Ah, just seen update All this honestly couldn't have happened at a worse time.

If you need some webspace on a dedicated server give me a shout. I know what its like so if you need to preview a website or similar drop me a line (email in trust)

 

[Izi]

Many thnaks for the offer. Have just emailed a client some screenshots, should be ok.

Stuff like this really makes you appreciate how reliant upon tsohost I am.

Me to... Not really sure what else can be done without huge expense.

The only thing I could suggest is to 50:50 spread your clients from different providers.

 

[Izi]

But on the other hand, using 2 supplies doubles your risk of problems so always a balance.

How do you mean? Wouldn't it half the risk. Say you have 50% of your customers with you and 50% with another company in a different data house then if the first server went down only 50% of your customers would be ringing you

Also, whats the latest on the UPS? Is the generator fully fueled?

 

[Izi]

I don't think a knee jerk reaction to move some of your customers elsewhere would be a great move. IMO of course.

I agree... I'm not going to, was 'just saying'.

I still want to know what Rob Kerry means by blue square being the Libl of data centers.

 

[Izi]

I guess he's insinuating that Blue Square is not one of the highest quality data centers.

I guessed that

Just wanted to know exactly what he meant. I.E what does Bluesquare lack that more modern data centers have?

 

[Izi]

Nothing, BSQ is only 4 years old and is light years ahead of other DCs in terms of facilities and service. It's a full Tier III datacentre, above what many hosts use including some rather pricey hosts.

Obviously something serious went wrong today however which will require a full review to decide what to do going forward. Power outages shouldn't happen.

Thanks Beansprout.

Do you know when we will be back on mains?

 

[Izi]

They're checking the UPSs to do the switch back, but given what the UPSs have done today I'm comfortable with taking a large can of diesel to the DC every morning....

(Note, that of course is a joke!)

I wonder if they will replace them or just re use them.