British Airways - Massive IT Disruption Worldwide

Soldato
Joined
3 Oct 2009
Posts
19,892
Location
Wales
My firm has outsourced their entire IT "department/system/infrastructure" (whatever you want to call it). We dont have a single IT person on site and therefore anything that can't be solved by turning off and on requires a call out.

Our systems have had more failures in the last 3-4 months than the 4 years prior. Our last episode was where our entire case management system and Internet went down and the nearest available engineer had to travel from Milton Keynes to North Wales to fix it... It's ridiculous
 
Caporegime
Joined
18 Oct 2002
Posts
26,052
Yeah but think of the bonus that the person who came up with that plan got for the cost savings! Of course those people tend to retain their own VIP support staff, or have people to use computers for them, so are completely unaffected by the changes.
 
Associate
Joined
1 Sep 2010
Posts
1,432
Location
Herts
Wow. I get stressed when the plane leaves half an hour late and I'm just flying solo for a business trip - dread to think how things are with you both. Best of luck getting it sorted.

I'm struggling to imagine how a problem like this can be "a power supply issue".
I've been caught up in this....

Have been on holiday to Riviera Maya. Was due to fly back to Gatwick from Cancun today. Had no idea what had happened until we overheard an American couple on the next table at lunch talking about it. We didn't receive any information email/ text/ phone) to advise what was going on. After trying to access flight status on their website for ages which wouldn't load, I expect because of the high usage it did load and said our flight was cancelled. Our hotel was fully booked and BA wouldn't cover the cost of it anyway so we got our transfer to the airport. Waited in line for nearly 4 hours as they tried to put everyone on different flights tomorrow and the day after.

It turns out we now have to stay a night in Cancun then return tomorrow and get an American Airlines flight to Dallas then get another AA flight from Dallas to Heathrow 6 hours later. Then we need to make our way back to Gatwick to collect the car.

Soz for the rant but it's a right mess of a situation....

Update -

Woke up 8am in Cancun. Got flight number 1 to Dallas which arrived at 2pm. Flight number 2 was due to leave at 7:05. So we waited around at a bar in the airport 4-5 hours. Everyone boarded flight number 2 promptly - all seemed well. Inflight entertainment and certain other electrics weren't working properly. We sat waiting on the plane for 2 hours for them to fix it. Eventually no fix and the problem was too dangerous to fly with so everyone off the plane and are now waiting for another plane to be taken out of the hanger and prepared for us. Current 10:20pm.
 
Soldato
Joined
3 Oct 2009
Posts
19,892
Location
Wales
Yeah but think of the bonus that the person who came up with that plan got for the cost savings! Of course those people tend to retain their own VIP support staff, or have people to use computers for them, so are completely unaffected by the changes.
Nope, they get just as affected as the rest of us! That day we had about 6 hours of being able to do nothing at all in the office, so that's about 20 people with an average hourly charge out rate of around £180 sat around twiddling thumbs.. Cost saving I'm sure :D
 
Soldato
Joined
31 May 2009
Posts
21,257
Update -

Woke up 8am in Cancun. Got flight number 1 to Dallas which arrived at 2pm. Flight number 2 was due to leave at 7:05. So we waited around at a bar in the airport 4-5 hours. Everyone boarded flight number 2 promptly - all seemed well. Inflight entertainment and certain other electrics weren't working properly. We sat waiting on the plane for 2 hours for them to fix it. Eventually no fix and the problem was too dangerous to fly with so everyone off the plane and are now waiting for another plane to be taken out of the hanger and prepared for us. Current 10:20pm.

If flight 2 is the one which ends up in the UK, then you've about 600 euros per person waiting in compo by the time of your arrival, make a claim against the airline who flies the second leg using resolver.
Cheap as chips and worth it for one days delay.
 
Capodecina
Soldato
Joined
30 Jul 2006
Posts
12,129
Latest theory I have heard is that a "power surge" fried BA's computers - I believe that surge protection devices are available at reasonable prices on many good websites ;)

From BBC Online:
BA blames a power outage, but a corporate IT expert said it should not have caused "even a flicker of the lights" in the data-centre. Even if the power could not be restored, the airline's Disaster Recovery Plan should have whirred into action. But that will have depended in part on veteran staff with knowledge of the complex patchwork of systems built up over the years. Many of those people may have left when much of the IT operation was outsourced to India.
I am absolutely certain that even the canteen staff in Mumbai will have MCSE & CCIE Certification.
 
Last edited:
Man of Honour
Joined
19 Oct 2002
Posts
29,508
Location
Surrey
If you used to work for BA and your phone was ringing right now, what would your day rate be?

I actually had something similar happen to me a few years ago although the outage wasn't on the same scale of course. I'd left a company a couple of years earlier but received a call from them one evening. They were in the middle of a major incident and my name was on a piece of mainframe code for which they needed to know how it worked. I worked for a competitor at the time but still offered to join their incident call. Without access to the code anymore there wasn't much I could do to help them as I'd written it about ten years earlier. But it still felt the right thing to do. I'd have charged them if any random person had called me but it was someone from my old team so was happy to help them out for a couple of hours.
 
Soldato
Joined
25 Jun 2011
Posts
5,468
Location
Yorkshire and proud of it!
Update -

Woke up 8am in Cancun. Got flight number 1 to Dallas which arrived at 2pm. Flight number 2 was due to leave at 7:05. So we waited around at a bar in the airport 4-5 hours. Everyone boarded flight number 2 promptly - all seemed well. Inflight entertainment and certain other electrics weren't working properly. We sat waiting on the plane for 2 hours for them to fix it. Eventually no fix and the problem was too dangerous to fly with so everyone off the plane and are now waiting for another plane to be taken out of the hanger and prepared for us. Current 10:20pm.

Sorry to hear that. I hope by the time you read this you're well on your way to where you want to be or already there. I can't say how glad I am that I'm not caught up in this.

If you used to work for BA and your phone was ringing right now, what would your day rate be?

I've actually been in this situation more or less a few times. Twice was just a case of me doing a couple of months work for the client, paid. I didn't really need the work on one of those occasions and on the other, I really didn't like the company at all. But in both cases, I had former team mates who would struggle to manage so I took the work on for their sakes. In the third scenario it was actually me proactively trying to help my replacement. I heard from my former manager that things weren't working and I immediately knew from the description what had happened. My replacement had found some commented out code and uncommented it thinking it would solve their problem. There had been a change in approach part way through the project and my part of the software could therefore function in two different ways. The programmer who didn't really understand what they were doing was basically just trying things out and finding something they thought might work, tried it out. Unfortunately, what they did was change it to something that worked but did the wrong thing. They then went into a death spiral of making everything worse and worse. They were very out of their depth and what made my blood boil was that before I left I had contacted them to offer to meet up before they took over and explain the project fundamentals and answer any questions which they declined. I also emailed them twice after they started and offered - freely and on my own time - to drop by and go over the code base and they refused. You might imagine they would be able to pick things up anyway or that "proper documentation" would have saved things. But I was the only programmer on that part of the project with nobody else to explain it to before I left and there's only so far you can go. Hence my repeated efforts to help train him on it. All rebuffed. I heard from my former manager there that he was going round blaming me for the problems and saying it had never worked despite it having been successfully in use for a year and doing what it was supposed to. Shortly after that he went on long-term sick leave with "stress". (Public sector employer). I am still angry to this day with them trying to blame me. I left because the wages were too low and it seems that the replacement followed the 'pay peanuts, get monkeys' principle to the letter. If you don't understand something, don't just keep changing things in the hopes that it will do what you want! That works for basic website layout (this person's background). It doesn't work for a larger, back-end project.
 
Caporegime
Joined
29 Jan 2008
Posts
58,898
If you used to work for BA and your phone was ringing right now, what would your day rate be?

well assuming you're somewhere else on say 600 a day and you'd essentially need to screw over that company and recruiter by dropping their contract then it would perhaps depend on the duration they need you for? If they only wanted you back for a week then it might not be worthwhile unless for silly money... for 6 months then I guess you'd probably chance it with something like 2 grand a day
 
Back
Top Bottom