Netflix to cut streaming quality in Europe for 30 days

Soldato
Joined
17 Jan 2016
Posts
8,763
Location
Oldham
We maintain thousands of peerings with most ISPs all over the world, and we collect performance telemetry from hundreds of millions of players, playing our game globally. Using data science, we can see exactly when things are good and when they're bad.

Who is 'We' ?

Your reply to me was to say UK ISP's are saturated, yet in this part of your post you agree with me, they aren't.

Almost all the UK ISPs are performing very well, with very little in the way of any ping spikes or packet loss at peak time - this tells me they still have capacity and are not congested yet, mostly. From our metrics, the only one which appears to be struggling is EE, all the others are doing very well.

However, that's only the UK - just about everywhere else is carnage;

I'm only talking about the UK. I have no information on any other country.

You have to understand how the system functions to understand why Netflix are doing what they're doing.

The problem Netflix has; they have so much capacity, they could easily overwhelm most ISPs if all customers started watching their content. If that happened and infrastructure started falling over because of Netflix (which it easily could), providers would probably start de-peering / rate-limiting / disabling Netflix content on their networks, to prioritise normal traffic and remove the congestion. Because remember - the customers of the ISP are not paying their provider anything for Netflix access - they're simply paying Netflix.

If there was a problem, even though you're admitting that UK ISP's are performing very well, then why doesn't netflix just introduce local caching (I don't use netflix so I'm not sure if they do cache or not)? So the program can temporarily download in the background as people watch it.

By volunteering to turn down the quality, and reduce the bandwidth, Netflix are simply making sure they don't end up 'annoying' ISPs to the point where they become forcibly rate-limited or disabled.. Lets face it - we can all still watch it anyway, they're just being careful - they're not stupid, they have some clever people working there.

So as you say no UK ISP is saturated, and netflix and others are just doing this on their own choice. So we agree.

A couple of interesting articles about this;

A Look at Monday’s UK Internet Traffic and ISP Speeds
https://www.ispreview.co.uk/index.p...ndays-uk-internet-traffic-and-isp-speeds.html

Yesterday was an important day for broadband ISPs in the United Kingdom because it was the first working Monday after the mass closure of schools, which has concentrated the demands of both home (remote) working and family internet traffic on to residential connectivity. But as expected the day was busy, albeit uneventful.

Over the past few weeks we’ve all heard the often scaremongering TV and newspaper reports, which have claimed that broadband in the UK would be “unable to cope” with the change in internet traffic as more people suddenly find themselves stuck at home. We’ve already explained – repeatedly – why such an interpretation, which fails to understand both the nature of the traffic change itself and how ISPs actually manage it, is so misleading.

Vodafone Says Fears of Network Overload “categorically untrue”
https://www.ispreview.co.uk/index.p...of-network-overload-categorically-untrue.html

The Chief Technology Officer (CTO) of mobile and broadband provider Vodafone UK, Scott Petty, has blasted mass media reports as “categorically untrue” for claiming that their network(s) won’t be able to cope with the increase in voice and data traffic due to the impact of COVID-19 (Coronavirus).
 
Soldato
Joined
28 Dec 2017
Posts
8,442
Location
Beds
If there was a problem, even though you're admitting that UK ISP's are performing very well, then why doesn't netflix just introduce local caching (I don't use netflix so I'm not sure if they do cache or not)? So the program can temporarily download in the background as people watch it.
Part of the point of adaptive streaming is that it only downloads what it needs. If people started getting whole episodes/films because they watch 5 minutes of something, that's more overall traffic for the internet.

I do agree that they should offer downloads by choice though - if you know you'll watch a whole series, why not prepare the top quality files overnight? They could still serve the DRM key in real-time.
 
Soldato
Joined
29 Dec 2014
Posts
5,778
Location
Midlands

You said ISPs aren't saturated, you didn't specify which country, you simply proclaimed that there isn't a problem, and the action Netflix is taking is unwarranted. I'm saying a great deal of ISPs are definitely saturated at certain times, because I can see they're saturated. A few UK ISPs are struggling, most aren't however almost all EU ISPs are struggling. And based on that Netflix probably won't bother to discriminate from country to country, they'll most likely just lower the quality slightly globally, to prevent them from adding to the problem - why is this so hard to understand?

Also, before you start arguing, I should point out, that Netflix run some of the most comprehensive and advanced telemetry and analytics of anybody in the entire world, so they probably have a fair idea of what they're doing, and it's probably been very well thought through.

If you think they're all wrong, and they're doing it because they're stupid - maybe you should apply to work for one of their engineering teams and give them a hand ;)
 
Soldato
Joined
29 Dec 2014
Posts
5,778
Location
Midlands
It's also worth pointing out, that ISPs will always put on a brave face and say everything is just hunky-dory, because they don't want people to move away from them. But these are quite frightening and unprecedented times for the tech industry.

To give you an example, we just added an additional 4Tbps of bandwidth to our network, but the biggest problem now - is getting equipment and getting people to install it. For example, all the optics are made in China, fibre patch leads, and specially made patch panels, line cards, bits and bobs - all come from China, and there's a bit of a backlog on some of it right now.

And case in point, we're building a few new POPs in the US right now, and in some cases we simply can't get people on site because of the lock down, or it's taking 3-4 attempts, or we have to use local smart hands to do installs. So even if I have money to spend on extra capacity - getting it into production right now, is more difficult than it's ever been.
 
Soldato
Joined
17 Jan 2016
Posts
8,763
Location
Oldham
You said ISPs aren't saturated, you didn't specify which country, you simply proclaimed that there isn't a problem, and the action Netflix is taking is unwarranted. I'm saying a great deal of ISPs are definitely saturated at certain times, because I can see they're saturated. A few UK ISPs are struggling, most aren't however almost all EU ISPs are struggling. And based on that Netflix probably won't bother to discriminate from country to country, they'll most likely just lower the quality slightly globally, to prevent them from adding to the problem - why is this so hard to understand?

Also, before you start arguing, I should point out, that Netflix run some of the most comprehensive and advanced telemetry and analytics of anybody in the entire world, so they probably have a fair idea of what they're doing, and it's probably been very well thought through.

If you think they're all wrong, and they're doing it because they're stupid - maybe you should apply to work for one of their engineering teams and give them a hand ;)

I'm not arguing. I don't know the situation in other countries ISP's. I was just saying that UK ISP's aren't saturated (though I probably should have made it more clearer that I mean only the UK), and give a couple of links showing stats and reaction from one of the ISP's calling it “categorically untrue”. If the British media are lying about the state of UK ISP's then its logical to think they aren't telling the truth in other countries. Though I'm not educated on other countries so I'm not commenting about them.

Why would an ISP be saturated when most nights people are streaming netflix, amazon prime, playing online games, downloading. If the Internet doesn't collapse on say friday nights then where is the logic it would collapse wednesday afternoon? Even if everyone is at home during the day it still won't have the same impact as evening use.

When you said 'We' who did you mean? Are you part of an ISP? Can you say which one? If you are from an ISP, or some official in some capacity, what do you make of the two articles I posted? They are saying the opposite to what you're saying.
 
Soldato
Joined
28 Dec 2017
Posts
8,442
Location
Beds
Why would an ISP be saturated when most nights people are streaming netflix, amazon prime, playing online games, downloading. If the Internet doesn't collapse on say friday nights then where is the logic it would collapse wednesday afternoon? Even if everyone is at home during the day it still won't have the same impact as evening use.
Many people are normally at the pub, at gigs, theatres, cinemas, in the park, restaurants, working late... All those people are probably on their sofa now.

When you said 'We' who did you mean? Are you part of an ISP? Can you say which one?

It's difficult to tell, we run one of the worlds largest online games and we're seeing a huge amount of additional players playing, around 40-45% more than normal. I'd suspect Netflix is seeing a similar increase, and I'd expect anything online entertainment/gaming wise, to be seeing 40%+ more usage than normal.
I'm a network engineer, spent most of my life working for ISPs and networking vendors (worked for Juniper for a while) I used to do a lot of core network stuff, but then later lots of Broadband work, mostly high scale BNG stuff. I ended up getting poached by the games industry, mostly good timing with a bit of good luck - now I help run a global low-latency eSports network.
:)
 
Soldato
Joined
29 Dec 2014
Posts
5,778
Location
Midlands
When you said 'We' who did you mean? Are you part of an ISP? Can you say which one? If you are from an ISP, or some official in some capacity, what do you make of the two articles I posted? They are saying the opposite to what you're saying.

I work for one of the worlds largest online game studios, and we have over 100 million people playing our games monthly. We collect telemetry from each player, cross-reference that with whichever ISP they're coming from, along with a whole load of other data science stuff, from that we can measure their performance by the second, and quality of experience, be it their ping, jitter (ping variation) or packet loss, disconnects and many other things.

Because we have such a huge cross-section of people connecting, we get a very good range of results that we can look at, and because our traffic is very sensitive to congestion, we're probably going to detect congestion before anybody else, because very few people have the level of insight that we have.

I don't really want to go into a deep technical dive about how broadband networks work, (unless you want me to). The main problem with the article, is that it's mostly talking about backbone and internet congestion (such as the linx graphs) the bottom line is, it's quite cheap and easy to add extra capacity in places like that, because in EU and NA - bandwidth is cheap. However - internally on the broadband side it's not so simple. Most ISPs build their networks in an oversubscribed model, in the sense that they oversubscribe the access-side of the network, rather than the external internet facing side, and it's here where the biggest bottleneck normally occurs, and it's a lot harder and more expensive to add capacity there.

I can tell you from our own telemetry, that most of the EU and some UK ISPs see a tremendous ping spike, around 6pm-9pm, where some of the pings go from 20-40ms, up to 80-100ms along with packet loss. Yet all of the peering links we have with those ISPs aren't even close to congested, they're only running at perhaps 40-70% used.

When the players complain, many of them are quite tech savvy and send us traceroutes in complaining - when we look at the traceroutes, we quite often see high-latency in the first 2-3 hops of the traceroute, which almost always points to a congested BNG node, (the ISP's broadband server) rather than congestion on the internet itself. If the BNG node is congested, then absolutely everything going back to all the broadband subscribers will be seeing some sort of impact to their performance.

On Vodafone - our metrics show Vodafone is handling it pretty well, and lets face it - their CTO is never going to come out in public and go "We don't have enough capacity, all our customers are suffering" is he? :D
 
Soldato
Joined
1 Mar 2010
Posts
21,875
A couple of interesting articles about this;

A Look at Monday’s UK Internet Traffic and ISP Speeds
https://www.ispreview.co.uk/index.p...ndays-uk-internet-traffic-and-isp-speeds.html
obvious question after reading that .. doesn't seem to discuss anything about whether packet types proportions have changed (eg. more real time games, this, supposed, remote schooling) that may demand better round trip times (ie. how is ping time incorporated in their discourse);
using a vpn system with a remote desktop to work .. it's usually the response time that makes or breaks it ... CAD type applications anyway.
 
Soldato
Joined
29 Dec 2014
Posts
5,778
Location
Midlands
obvious question after reading that .. doesn't seem to discuss anything about whether packet types proportions have changed (eg. more real time games, this, supposed, remote schooling) that may demand better round trip times (ie. how is ping time incorporated in their discourse);
using a vpn system with a remote desktop to work .. it's usually the response time that makes or breaks it ... CAD type applications anyway.

It's a good point, there are more people playing online games right now, than I've literally ever seen in my entire life, a frightening amount. When you mix all of those small packet sizes in, with other real-time apps, such as all the additional video calls, VPN, traffic, etc, etc, it's really going to be stressing those queues and schedulers on those broadband routers.

On the plus side, a few weeks ago it was taking almost 20 mins to get a game of Overwatch, now at 11am this morning it was taking 2mins to get in a DPS queue :D
 
Soldato
Joined
29 Dec 2014
Posts
5,778
Location
Midlands
I know how the basic Internet works. But you obviously have far more details than me and what was provided in the links I posted, so I'll metaphorically bow to your more knowledge :)

s'cool man :)

We actually publish some of our statistics using something called lagreport, where it'll give your internet connection a score based on your connectivity to our servers, but it'll also compare your connection to all the other players in your area, if you scroll down to the bottom - you can compare most of the different ISPs in your area, and select score/ping / packet loss / jitter, to see how things look at certain times;

You don't need an account to make it work, just accept the thingy

https://lagreport.euw.leagueoflegends.com/en/

Here in IRL, if I compare my connection (Virgin Media) to Three, at peak time, we can see Three suffers from quite a bit of congestion, as we see packet loss at peak times, which only started happening in the last few weeks.

o6ZLCa3.jpg
 
Soldato
Joined
1 Mar 2010
Posts
21,875
Last edited:
Back
Top Bottom