The internet just imploaded!

Mr_Orange

Mr_Orange

Mr_Orange

Associate
Joined
5 Jun 2006
Posts
1,152
Just had a weird thing happen.

Internet services at the office went down. Nothing too weird with that but I got it working again and logged onto an few IRC channels usually filled with thousands of people from all over the world and there were 10 people logged on!

Did we just have a world wide internet blip?
 
Yeah for some reason BBC/Facebook/Sky Sports works

But all my MSN/Live Service & some other services/sites aren't working

Some signatures aren't even loading on here now.
 
Last edited:
If anyone cares it appears to have been a malformed AS path issue at the moment, spiked CPU on a good few routers around the place. I gather BT lost several dozen 622mb central pipes at least (and a bunch of higher bandwidth circuits) and some of the IX's dropped 100Gbit of traffic.

This is going to be fun to track down...
 
bigredshark said:
If anyone cares it appears to have been a malformed AS path issue at the moment, spiked CPU on a good few routers around the place. I gather BT lost several dozen 622mb central pipes at least (and a bunch of higher bandwidth circuits) and some of the IX's dropped 100Gbit of traffic.

This is going to be fun to track down...
Click to expand...

That kinda explains things I guess means nothing to me but good to see something about it at least!

Good luck with whatever you are doing lol :)
 
bigredshark said:
If anyone cares it appears to have been a malformed AS path issue at the moment, spiked CPU on a good few routers around the place. I gather BT lost several dozen 622mb central pipes at least (and a bunch of higher bandwidth circuits) and some of the IX's dropped 100Gbit of traffic.
Click to expand...

BT NET described it as a 'total network loss for 7 minutes' in their email back to us. Lost a thousand or so trader connections so it wasn't a terribly fun morning :(
 
I just looked at the steam active users graph and have a look at this :D
capturejgt.png
 
Duke said:
Whats that in English? Understood the rest of it bar that :p
Click to expand...

As basically as possible - when routing information for IP addresses is propagated round the internet it has a field called an AS Path attached, which details the sequence of networks that information has passed through. It's important to how the internet works.

Now when this field is incorrectly formed or very long, that can cause CPU on core internet routers to spike up to 100% - which appears to be what happened here.

Most people filter the updates and reject any which appear suspect but the filters aren't foolproof (though that said, our network was alright as my filters did reject the updates so it was possible to block...). It does come as quite a shock to most people to realise how fragile the internet can be for all it's resilience...
 
You must log in or register to reply here.
Back
Top Bottom