My 2cents as a Network Engineer with more than 20 years experience.
I worked for one of the large west coast game studios for 4 years, we ran our own data centres and spent in the range of $200-300M on building our own global, low-latency network for gaming. At the time it was the only option available to us, if you wanted to deliver a zero packet-loss, low ping experience to players - you simply had to have your own infrastructure. Routing player traffic over the internet, via paths which change from day to day, over congested IX links and crappy transit providers, does nothing but deliver a rubbish experience, to cap it all off - when you get DDOS'd - the provider simply disables your links to the DC, to protect their other customers.
We purchased extremely expensive low-latency routes, things that no ordinary company would buy because it wouldn't make business sense - but with the company ethos revolving around 'player experience above everything else' and we could easily afford it, it was worth it. It was a big success.
At the same time, despite being pretty good at running a network - for a whole host of reasons we were not so great at running data centres, as a result the business decided that we'd start moving the servers into AWS, This was good and bad. It was good because it's almost too easy and getting something going would be a matter of minutes or hours instead of having to wait for servers to be purchased etc. It was bad because developers - who had no knowledge of how to build infrastructure, would simply spin things up all over the place inside AWS, without any real design knowledge, or knowledge of how to build resiliant systems - things would break for silly reasons and we ended up with a virtual birds-nest of things.
At this point our global network was routing traffic to AWS servers via AWS directconnect - a feature that allows you to directly connect your own infrastructure to AWS in the region where your instances are. Because AWS was only in limited regions, AWS's own peering and transit connectivity sucked compared to players being routed directly into the AWS region via our own POPs - which were all rigged for low latency and as close to the players as possible.
The big 'alarm bell' rang for me, when AWS released a feature called 'global accelerator' essentially - this allows a customer, to spin up their services in any region they like (or all regions if they wish) and traffic can be routed via 'anycast' from anywhere, to whichever region is closest. You can then ride AWS's own backbone, to get data from other regions - essentially you're using AWS's own internal infrastructure where you'd normally use your own WAN or MPLS network.
The fateful date eventually came, when the business decided to do a comparison between our network, vs AWS's global accelerator. At first we thought that our specially crafted network, designed entirely for one thing (64 byte packets, pings of <30ms and zero packet loss) would make a mockery of AWS and we had nothing to worry about. We made changes to the game clients, so that when the system ran the matchmaking loop, it would 'ping' both networks, ours and AWS's - then put you on whichever has the lowest ping.
In an instant, literally 50- 60% of all the players were flipped to the AWS side. In some areas almost all of them flipped to AWS, the pings were the same or lower and crucially the experience was the same or better. Some areas we did beat AWS by quite a margin, but they were more remote regions with fewer players - or areas which are an absolute nightmare to build connectivity where the cloud providers aren't very well established yet (South America)
At this point the die was cast, despite the full AWS service being hellishly expensive - developers could now build a game, build the infrastructure for it and connect players to it, without them having to work with anybody internally. We ended up having conversations like this;
"We want to turn on this stuff on the network, what do we need to do" "You need to do this short list of things" "oh ok - if it's going to cause issues, we'll just turn it up in AWS' people naturally take the path of least resistance, and there was no getting around that fact.
I realised at this point that the future is limited for companies which have their own infrastructure. AWS is very expensive, but it's priced very cleverly. Amazon know exactly how much it costs to build infrastructure, they know exactly where the pain points are, and once you have that convenience and you've tasted it, it's very difficult to go back. Bottom line - if we were running a global e-sports title now, in 2021 - we wouldn't entertain the idea of building any private infrastructure. With $100Bn and a team of 1000 engineers, I couldn't design a network that would beat AWS - nobody can, it's a pointless endeavour.
I've watched a number of companies disappear recently, one notable example was a startup which around 15 of my friends moved to (specialised in low latency gaming traffic) - at one point they were carrying most of the traffic for Fortnite in Asia and a few other places, because AWS was crap. Then Amazon decided to tweak their routing and do some optimising, overnight all the traffic flipped to AWS and the entire company went bust.
From a career perspective, I wasn't really sure what the future held for me - and then AWS reached out to me and offered me a senior engineering role, designing next-gen IP fabrics, routing and automation, which is where I am now... My options are far more limited than they were say 5-10 years ago, when everybody was building their own MPLS networks and stuff. Those problems have simply gone away, the cloud providers have become so large, have such good infrastructure that you'd be a madman to build any sort of network unless you absolutely had to.
Cant speak for anyone else but if i try and get a sign off for a new server it takes a real effort getting a business case signed off but cloud solution its blank cheque time even if its not the most cost effective.
I've witnessed this many times, and it speak volumes as to where people are really willing to spend their money. At the game studio I mentioned, the company struck a deal with AWS (we were a top 10 AWS customer, and had a seat at their table) where we comitted to spent a frankly insane amount of money, over three years. The end result was that you could literally spend
any amount of money on AWS and the bill would just get added to the 'tab' and disappear until the end of the year when we'd get the AWS bill; a bill that ended up being on the high end of 9 figures..... Yet - if I wanted a *small* amount of money, such as $250-500k to equip a new POP, I'd have to go through the eye of a needle, it sucked.
that would cover the budget of company i work for a few years! 