The network engineers thread

[Felon]

So who here is a network eng? I assume we must have more than a few on here, thought it would be nice to have a friendly thread dedicated to the craft

Whether it's work related, studying certifications, or just curious - thought it would be a good place to chat.

 

[Felon]

I get surprised and saddened by the lack of fundamental knowledge by ‘network engineers’ these days.

Yeah, it's hard work trying to find good people at the moment, I have 2x roles for senior data centre engineers open in my team at the moment (lots of Mellanox/Nvidia), but to be fair - I'm looking for fundamental knowledge more than specific vendor experience, and finding good candidates at the moment seems almost impossible.

I have a long history or service-provider, but now I mostly do Data Centre network engineering these days and it's a lot more challenging than I though it would be, modern switches have gotten so much more efficient in power, so they're much more limited in features and have no buffers - yet what I'm asked to do with them by the business is a constant challenge:

"Can we plz stretch layer-2 vlans across 20x sites?"

"Why do the switches each cost $12000, can we not just buy netgear ones from Currys?"

"The network sucks, if I run my 25G ports at 100%, I get packet loss, omg fix it"

 

[Felon]

But you said you’re a network engineer.

Around, 25 years now? 2x CCIEs (RS/SP) and a whole load of other expired crap (lots of Juniper, but couldn't be arsed to do a JNCIE)..

If you can do Cisco, you can do like 95% of everything else - especially Cumulus (which I think has one of the best BGP implementations ever), it uses FRR (Free range routing) which has an IOS cloned CLI. The only one I'd say Cisco-centric engineers struggle with, is Nokia - I remember first working on Nokia and feeling like I was in some other alternate reality of pain - I've never used a more horrid interface. That said - Nokia stuff is really meant to be used with their full automation ecosystem - at which point it's great.

BTW don’t get me started on L2 stretches!

Application team be like; "We hard coded the IP addresses into the application we wrote, and we can't change it, why is the network so crap?"

 

[Felon]

(everyone hated Broadcom even in 2016/17, even more reasons now!)

Yeah the situation with VMware is mad right now, we’re ditching them - so many others are too…

I think our price went up by 4x or something? I’m not really sure what they’re doing…

Although Broadcom’s newer ASICS are decent, I’m just on the tube headed to the DC to find a large pallet of Arista 7060X5 eval kit, based on Tomahawk 4,

 

[Felon]

Took a delivery of a huge load of eval kit today around $1M worth (list price), including a large delivery of 400G-OSFP-DD DAC cables, they're absolutely huge.

Standard 25G SFP on the left, 400G-OSFP on the right, it's like pushing something the size of a mars-bar into the the router

 

[Felon]

Cut my teeth on Catalyst 6509 switches and 7206Vxr routers.

7206VXR.... Now there's a good box!

There was a point in time - would have been what, sometime around 2001-2005? where those things were literally everywhere, with an NPE-G1 you could do literally anything with them.

 

[Felon]

I find Cisco just too expensive these days, and their DC portfolio sucks pretty bad, especially compared to serious DC players like Arista and Nvidia/Mellanox.

That said, up until the Nvidia acquisition - you could get a Mellanox 3700 for like $10-12k list, since the Nvidia takeover - you're now looking at $30-40k list, they hiked those prices big time.

That said, the Cumulus BGP implementation is one of the best there is for clos/spine-leaf topologies. The only problem I have with Nvidia right now, is that they're going 100mph down the whole GPU route, if you want features adding to the software and it doesn't involve filling the DC with GPUs, forget it.

 

[Felon]

Goal one day is an IE (Cisco or Juniper) and work in professional services rather than a mix of managed service and support but we shall see.

I spent a few years doing consulting work for Juniper, via an elite partner/reseller - to be honest it was great, I got to work with really good customers who knew what they were doing an just needed some help here and there (testing, design, etc). Then I'd end up on the polar opposite - customer's who'd purchased a heap of MX or PTX, but didn't have a clue what they were doing, and I'd end up in a hostile environment trying to help people who didn't want to be helped

So it was cool, as I got to see the best of both worlds - was interesting.

 

[Felon]

But then again, 400Gbps is some serious business - what sort of device(s) do these DAC cables go into?

Those currently connect into Arista 7289R3 edge routers, which I'm using as a DCI interconnect: https://www.arista.com/en/products/7280r3a-modular-series-quick-look (currently evaluating the whole environment)

We did look at Juniper PTX10k, but Juniper were being silly with their pricing ($300k per box), so we stopped talking to them

Those gigantic 400G optics (OSFP), are actually the older standard of 400G (it uses 8x50G lanes to get the 400) and they run really hot (which is why the optic is so much bigger)

The newer version (QSFP56-DD) are smaller, cooler and run 4x100G lanes, they're what we're using in the spine layer.

Only problem is, you can't connect 400G OSFP at one end, and 400G QSFP56-DD at the other, as the SERDES both operate at different rates (one end is 8x50g the other is 4x100) so you need to terminate on different line-cards, which is a headache.