AMD Epyc has problems when you max out PCIe lanes

[Quartz]

Linus Tech Tips has an interesting video about the problems they had when they maxxed out the PCIe lanes on their Epyc server with umpteen NVME drives

TLDR There are major performance issues; it's all too fast and the bandwidth is overloaded. I'm wondering if Intel solutions have the same problems?

 

[Quartz]

So perhaps more RAM might have solved the issue?

 

[Quartz]

You need to watch the first part of this build video to understand what is going on.

I did watch it and was fully cognisant of them fully using the PCIe lanes.

Put simply they built a system with Mass Storage that is faster than RAM.

As I said in my OP, it's too fast.

 

[Quartz]

More to the point.. 24 nvme in raid 0, who takes that sort of risk in the real world?

I thought they put on ZFS and RAID 5 or something, not RAID 0.

 

[Quartz]

They even upgraded it to a 64c CPU and changed to a 32c CPU in the end.

I wonder if reducing the number of cores would have worked.

 

[Quartz]

Why you try to spin it as "EPYC problem"?

Because that's what it appears to be.

And from the above post clearly you didn't understood the whole video

Are you sure? One of the highlighted problems was interrupts and processes having been switched away and the solution was to switch interrupts off and move to polling. Managing umpteen processes via polling is difficult and, as LTT found, CPU-intensive. So by reducing the number of cores you obviate that. Each process / core has more to do, of course, but it's just shifting stuff between the NIC, RAM, and the SSDs. You could do it as three processes: one moving data between the NIC and RAM, one process moving data between RAM and SSD, and one master process. That's a lot simpler with much less overhead.

 

[Quartz]

You missed the point that they are using ZFS with raid redudancy. It must hold 3 copies of the data in the memory while it writes them.

No, I didn't miss it.

I've always considered them to be a bit of let's see what happens and have a laugh.

I agree. There's a very Clarkson-era Top Gear element to it.