'Olympic' outage in East London

Andrew_McP

Andrew_McP

Andrew_McP

Soldato
Joined
21 Apr 2003
Posts
3,350
Location
South North West
Someone digging foundations for some Olympic white elephant or other was careless with a big drill on Saturday. Took out a BT tunnel 32m down... nice and safe, you'd think. :-)

http://noc.enta.net/2009/04/outage-framestream-leased-lines/comment-page-1/#comment-15783

http://forum.o2.co.uk/viewtopic.php?t=17231

The last couple of days have been a great advert for broadband dongles... though I wish I'd kept the tenner a month '3' dongle I set my mother up with. This Vodafone dongle's better financially (£15 for a Gb without time limit, great for emergencies like this) but the performance here is definitely inferior.

Andrew McP
 
Last edited:
Indeed, we've lost an entire BT central pipe into our DSL platform from this incident so we're running at very high load now on the other centrals. We're not getting much more information out of BT than what's above really - they're talking a couple of days before service restoration as a worst case.

I'm surprised no other ISPs are affected as this pipe fed a central in telehouse, where I imagine more than a few ISPs have their centrals.

Still, just one of those things and we're running without any problems on the spare capacity we keep available for exactly this reason, load's a little high but i'll live with it for a few days...
 
It's still what you'd describe as an enormous mess...

Restoration Summary at 10:30
• Second fibre cable successfully completed at 08:00
• All 75 concentrators impacted have now been restored.
• No PSTN customers out of service. Severe congestion due to low capacity. 999 services restored.
• Re-routes – 52 – re-routes suspended due to systems failure (escalated resolution)
• GS – BSkyB fault due to another fibre issue ETR 14:00. 363 faults reported, 17 confirmed restored, although expect more to clear as customers arrive this morning
• BTW – 456 faults ( down from 506). 238 cell sites still affected ( down from 283 )
• BTR - 746 Bus faults, 3,569 res faults
• All Met Police ccts to all three sites have been restored.
• All urgent Airwave ccts have been restored.
• Government circuits have been restored.

Action Plan and Progress
• High level Plan (cables are numbered in order of priority)
• Action 1 - Identify damaged cables and ccts and establish priority plan – complete (fluid)
• Plan A – Stage 1 Openreach to provide new street level cables
&middot Provide 3 x new 276 fibres at street level. Complete.
&middot Provide 4 x new 276 fibre cables at street level. Not yet pulled in.
&middot Priority cable 1 complete and working
&middot Priority cable 2 complete and working
&middot Priority cable 3 ETA 17:00 today
&middot Priority cable 4 ETA 19:00 today
&middot Priority cable 5 ETA – being reassessed
&middot Priority cable 6, 7 and 8 – no ETA
&middot If all target completion times are met today, 30% of the damage will be restored.
• Plan B – Re-route using all existing spare fibres.
• This is nearly complete but we are still finding new fibres and using them to re-route services as they become available.
• Plan C – Provide and utilize PDH and SDH radio systems if required
• PDH unit on site. SDH unit may not be available, checking options – no ETA
• Plan D – Provide Sat Comms solution if required
• Establishing the viability and suitability of this option. Unlikely to be viable
• Plan E – Restore damaged copper cables (thought to be only 1 live) – no ETA
• Plan F – Repair tunnel and cabling – no ETA

NB Plans A and B are active and running in parallel. Indicative target time for all damaged cables to be off-loaded – late 9th April (this is not a commitment, only a target)

Other Key Issues and Updates at 07:50

• Cumulative number of fibres repaired – 192 or 10.2% (up from 96 / 5.2%)
• Cumulative number of 2 Mb/s equivalent circuits restored – 6995 or 15.2% ( up from 1883 / 6% )
• In total there are 1,836 fibres are to be repaired. If we have no further delays, we expect to have 30% of ccts recovered by CoP today. Current expectation is that the repairs will run through to Thursday.
Click to expand...

10.2% of circuits restore, a nice way of saying 89.8% are still borked.
 
I thought I read somewhere they hoped to have it all done by the weekend - I thought they meant last weekend, so I am guessing it was meant to be this weekend!

Looong old outage!
 
Man. I would not want to be working in their NOC over this outage... Or be one of their provisioners...

Still might mean more business for us :D Hehe.
 
Wondering why I was getting text messages from O2 regards my broadband (staying at parents using 3g dongle) so have not used my broadband at my place.
 
malef!c said:
I thought I read somewhere they hoped to have it all done by the weekend - I thought they meant last weekend, so I am guessing it was meant to be this weekend!

Looong old outage!
Click to expand...

BT updates are saying virtually everything will be restored this evening at present, they've dumped a load of fibre in surface level ducts to work round it and they're just splicing now (though 1800 odd fibres to splice is a hell of a lot...)

• Priority cable 1 complete and working
1 Priority cable 2 complete and working
2 Priority cable 3 ETR - 07.00 - pushed back from midnight tonight due to crossed fibres
3 Priority cable 4 ETR 19:00 08/04 - pushed back from noon 08/04 due to crossed fibres
4 Priority cable 5 ETR – 07:00 08/04.
5 Priority cable 6 ETR – 07:00 08/04 - improved from 19.00 08/04
6 Priority cable 7 – no information available
7 Priority cable 8 ETR – 19:00 08/04 - estimated
8 Priority cable 9 ETR – 19:00 08/04 - estimated
9 Priority cable 10 ETR - 19.00 08/04 - estimated
10 Priority cable 11 ETR - 19.00 08/04 - estimated
• Plan B – Re-route using all existing spare fibres - continues.
• Plan C – Radio links – set up for C7 signalling - in place
• Plan D – Satellite Comms – not viable
• Plan E – Restore damaged copper cables- further checks being made
• Plan F – Repair tunnel and cabling – no date available
 
I suppose if anything this is a perfect advert for having a more redundant network infrastructure. I can't believe one civils even can affect so much with no redundant fibre routes.
 
pdw8 said:
I suppose if anything this is a perfect advert for having a more redundant network infrastructure. I can't believe one civils even can affect so much with no redundant fibre routes.
Click to expand...

Depends on the level your talking about. For ISPs to have more resilient networks and multiple routes - very definately, I'm still gloating about the ease with which our network shrugged off the effects and kept going, it's a perfect advert for having well designed networks with resiliency.

For BT I disagree, they'll give anyone who'll pay diverse connectivity and it was just unfortunate to loose such a major cable tunnel, I'm a big critic of BT normally but they've handled this well and kept us up to date. I can't really fault them.

From our point of view it's an event equivilent to a major POP getting blown up or something, sure we can reroute round it for traffic transitting it but the connections which run originate or terminate there are going to be affected. No way round that...
 
Our network has had no impact except a circuit used for VOIP. We dont use BT directly for any customer circuits, mainly COLT, Expo-e and Hso now.

You would think that something like this should not affect critical services, Governemt or even TfL though.
 
pdw8 said:
Our network has had no impact except a circuit used for VOIP. We dont use BT directly for any customer circuits, mainly COLT, Expo-e and Hso now.

You would think that something like this should not affect critical services, Governemt or even TfL though.
Click to expand...

Yeah, a little disturbing that tfl don't have a little more resiliency but then again it's only live management of traffic lights that's stopped working, is that capability worth the extra chunk of cost resiliency would have cost.

Interesting we didn't loose any LES circuits through it, we have loads in the area so I'm surprised. Relevent as a lot of our tail circuits for last mile access are BT (which I guess is relevent to your providers point, I know expo-e use BT circuits a fair bit for last mile...).

And just to confuse things a bit more, while we lost a BT central connection. The BT IP Exchange circuit for SIP trunks in the cab next door didn't blink. BT's network is a complex beast...
 
You must log in or register to reply here.
Back
Top Bottom