From data center to main internet(bandwidth)

jsmoke · 3 Mar 2017 at 14:15

Anyone know how the internet can handle the bandwidth coming from a DC or server farm, take a FB SF for example, 100' s of thousands of servers and their own private optical network onto the local loop of the internet. Each farm much traffic a massive about of bandwidth hence their own optical route will be tailored to cope and will be extreme. However the main internet cabling system must be many times capable of handling much more bandwidth than a SF needs, is this correct, is it just a case of laying many more optical fibre in the ground or how does it all work?

How much bandwidth can the main internet fibre system handle.

ChrisD. · 3 Mar 2017 at 14:20

Not sure what you're asking?

The cable between the UK and US is about 320Gbps the last I knew, ISPs can transfer higher than that.

tom_e · 3 Mar 2017 at 14:29

https://en.wikipedia.org/wiki/Internet_backbone

Sp00n · 3 Mar 2017 at 14:30

The infrastructure has multiple paths to every location and every path has multiple links.

400G links being rolled out by Telia at the moment in europe.

BigT · 3 Mar 2017 at 15:45

tom_e said:
https://en.wikipedia.org/wiki/Internet_backbone

This made me laugh in that article:

for example, in 2011, a woman in Georgia pierced a fiber backbone line with a shovel and left the neighboring country of Armenia without Internet access for 12 hours

Rroff · 3 Mar 2017 at 15:51

Not overly familiar with it but I think most traffic by its nature tends to be fairly bursty and/or spread out over times of day, etc. if all servers were (potentially) running flat out saturated then there simply wouldn't be enough bandwidth available on the links and things would slow down. I believe also when interconnects are put in there is a certain amount of "dark fibre" etc. which can be brought online over time as/if the demands grow to require it.

joeyjojo · 3 Mar 2017 at 16:06

Read Tubes by Andrew Blum (yes it's a meme title

)

https://www.goodreads.com/book/show/13036199-tubes

Very interesting, light book about one man's research into 'where is the internet?' Holiday reading really but full of well researched stuff and he visits all the places he talks about (data centres in the USA, intercontinental cable landing spots, the Amsterdam Internet Exchange, etc.).

jsmoke · 3 Mar 2017 at 16:27

joeyjojo said:
Read Tubes by Andrew Blum (yes it's a meme title )

https://www.goodreads.com/book/show/13036199-tubes

Very interesting, light book about one man's research into 'where is the internet?' Holiday reading really but full of well researched stuff and he visits all the places he talks about (data centres in the USA, intercontinental cable landing spots, the Amsterdam Internet Exchange, etc.).

Looks good, think I will pick this up, thanks.

nutcase · 3 Mar 2017 at 17:58

10Gbps backbones are standard fare for an ISP now, with datacentres having multiple nodes. As others have said, where there are trunks they are measured in hundreds of Gbps

jsmoke · 3 Mar 2017 at 18:03

What kind of device would connect say a DC or SF to the main internet, in other words where to the cables(or fibers) meet to be routed, are the fibers physically laid direct to an exchange or are there routers/boosters dotted around the place in security cabinets.

ChrisD. · 3 Mar 2017 at 18:19

Cisco ASR 9000s or Juniper equivelents for ISPs, look up core routers for larger networks.

asim18 · 5 Mar 2017 at 03:20

jsmoke said:
Anyone know how the internet can handle the bandwidth coming from a DC or server farm, take a FB SF for example, 100' s of thousands of servers and their own private optical network onto the local loop of the internet. Each farm much traffic a massive about of bandwidth hence their own optical route will be tailored to cope and will be extreme. However the main internet cabling system must be many times capable of handling much more bandwidth than a SF needs, is this correct, is it just a case of laying many more optical fibre in the ground or how does it all work?

How much bandwidth can the main internet fibre system handle.

Not enough. Only 9.9% of requests for facebook photos are actually served by their "main" data centres.

Facebook, youtube, etc. employ vast content delivery networks. Only a small portion of total traffic is handled by "main" data centres. The content itself is served from cache/CDNs.

https://code.facebook.com/posts/220956754772273/an-analysis-of-facebook-photo-caching/

Basically if someone uploads a selfie, it will be uploaded to the main datacenter servers. But when 100 people from london view the photo, it will only download from the main data centre once and be held in a cache server somewhere more local, so the other 99 requests from London wont hog any resources at the main data centre.

Same happens with youtube videos.

jsmoke · 5 Mar 2017 at 10:55

Amazing how it all works in harmony like that, some smart engineers out there and nobody knows who they are, they don't get any credit. If only Joe public knew some of what goes on behind the scenes to make their lives more comfortable..

Caged · 5 Mar 2017 at 10:58

The people who do it all will frequently blog about it and present at industry conventions.

https://code.facebook.com/posts/