AMD Talks Next Generation Coherent Interconnect Fabric Connecting Polaris GPUs, Zen CPUs and HPC APU

[AlamoX]

AMD’s Coherent Fabric To Clock in 100 GB/s Interconnect Speeds – Open Nature Design


in a perfect situation, all GPUs inside a workstation can be made to act as a single resource that aims to solve a specific set of problems. Currently, AMD only provides a software based multi-GPU framework added in their recent Boltzmann Initiative but moving forward, AMD knows that an update to their multi-GPU solution, also known as CrossFire, is needed to provide a hardware based multi-GPU solution that aims to deliver a robust scaling across AMD’s GPUs. For this matter, they are developing their own open Interconnect fabric that will not only allow AMD GPUs to run in tandem with one another but also allow third-party solutions (GPUs, FPGAs).

PCI-Express is already seen as a bottleneck when connecting several nodes in high-performance sectors. AMD sees their current PCI-e and CrossFire solutions not working with next generation machines hence they have to design a new coherent fabric. The interconnect will offer speeds of 100 GB/s across multiple GPUs and APUs that are featured inside AMD powered compute machines and will deploy some open standards. Asking if the interconnect will also maintain memory coherency and sharing between the GPUs and CPUs, Raja stated that he can’t reveal that right now but will definitely have a detailed showcase of their coherent fabric later on as coherency between their several chip designs is being kept in mind.

you can find the full Article Here

 

[Mauller]

AMD 'Coherent Data Fabric' - Their next gen interconnect tech.

AMD Has officially revealed their next Gen Interconnect technology, set to supplant PCI-E with compatible devices.

Oritigan article Link

IMG]http://cdn.wccftech.com/wp-content/uploads/2016/01/AMD-Computing-Platforms_1-635x357.jpg[/IMG

AMD Developing Ultra-Wideband, Low Latency Coherent Interconnect Fabric To Rival NVIDIA’s NVLINK

Since the creation of RTG (Radeon Technologies Group), Raja Koduri has put a lot of focus back to GPUs. In 2012, AMD’s strategy was quite different to what it is today. The company focused on their APU (accelerated processing units) and SOC (System-on-chip) designs however that didn’t work well for the company. The GPU department on the other hand has been a main driving force behind for them and that’s where both AMD and RTG is specifically eyeing at after the restructuring of the company in September 2015.

According to Raja, there’s a constant rise in compute demand for the high-end PC market. Specifically speaking about compute performance where the GPU clearly shines against modern CPUs, there’s just constant demand for greater computing needs in the high-performance computing sector which consists of HPC, Servers and the Workstation machines. To feed that demand, AMD is coming back with a big bang in the GPU department with their Polaris GPU architecture but their aim doesn’t stop at GPUs alone.

http://cdn.wccftech.com/wp-content/uploads/2016/01/AMD-Computing-Platforms_3-635x357.jpg[/IMG

[B]AMD’s Coherent Fabric To Clock in 100 GB/s Interconnect Speeds – Open Nature Design[/B]

PCI-Express is already seen as a bottleneck when connecting several nodes in high-performance sectors. AMD sees their current PCI-e and CrossFire solutions not working with next generation machines hence they have to design a new coherent fabric. The interconnect will offer speeds of 100 GB/s across multiple GPUs and APUs that are featured inside AMD powered compute machines and will deploy some open standards. Asking if the interconnect will also maintain memory coherency and sharing between the GPUs and CPUs, Raja stated that he can’t reveal that right now but will definitely have a detailed showcase of their coherent fabric later on as coherency between their several chip designs is being kept in mind.

[IMG]http://cdn.wccftech.com/wp-content/uploads/2014/08/NVIDIA-NVLINK-2-635x347.jpg[/IMG

Comparing to NVIDIA’s NVLINK which is going to have interconnect speeds of up to 200 GB/s, the AMD solution does have a slight advantage as it works across x86 processors as well while NVIDIA’s NVLINK works across the NVIDIA based GPUs and IBM’s Power CPUs. NVIDIA is already deploying NVLINK in two next generation super computer so it will be interesting to see AMD’s own solution going in action to power some high-end spectrum devices.

[/QUOTE]

[B][COLOR="Orange"] ** No Hotlinking **[/COLOR][/B]

 

[Mauller]

This should be interesting as from what i was reading before, it works over current PCI-E lanes. But each lane runs at a far higher Bandwidth.

This means that fewer lanes are needed when connecting to a GPU freeing up more Data lanes from the cpu to the rest of the system.

 

[humbug]

the AMD solution does have a slight advantage as it works across x86 processors as well while NVIDIA’s NVLINK works across the NVIDIA based GPUs and IBM’s Power CPUs.

That should make things interesting.

 

[nashathedog]

So they're saying that there bridgeless crossfire (XDMA?) is not capable of doing the job it currently manages with the 28nm chipped cards.

There's a lot that really bothers me about the current Fiji cards, Another example is they state they have a couple of techs working full time on memory management for Fiji, That's obviously only a temporary measure so once they introduce the next gen cards will it continue? Will it be a relevant requirement for next gen HBM cards or will HBM1 owners become neglected.
I really like my Fury Tri-x as Sapphire did a bang up job but has it got a lifespan ticking away a lot faster than usual?

 

[CAT-THE-FIFTH]

So they're saying that there bridgeless crossfire (XDMA?) is not capable of doing the job it currently manages with the 28nm chipped cards.

There's a lot that really bothers me about the current Fiji cards, Another example is they state they have a couple of techs working full time on memory management for Fiji, That's obviously only a temporary measure so once they introduce the next gen cards will it continue? Will it be a relevant requirement for next gen HBM cards or will HBM1 owners become neglected.
I really like my Fury Tri-x as Sapphire did a bang up job but has it got a lifespan ticking away a lot faster than usual?

The new fabric is a more a replacement for Hypertransport and AFAIK it is called "Freedom Fabric" and was derived from tech that SeaMicro had.

 

[Confused Stu]

So they're saying that there bridgeless crossfire (XDMA?) is not capable of doing the job it currently manages with the 28nm chipped cards.

There's a lot that really bothers me about the current Fiji cards, Another example is they state they have a couple of techs working full time on memory management for Fiji, That's obviously only a temporary measure so once they introduce the next gen cards will it continue? Will it be a relevant requirement for next gen HBM cards or will HBM1 owners become neglected.
I really like my Fury Tri-x as Sapphire did a bang up job but has it got a lifespan ticking away a lot faster than usual?

This is aimed at the HPC and high end server brigade, mate, rather than gaming PCs. Given every test I've seen has shown minimal losses when Crossfiring between PCI-E2 16x and 8x, I think it's going to be a good few generations before PCI-E3 connections are the limiting factor is a games machine.

In the short term, pretty sure this article is only of direct interest to Kaapstad with some of his setups!

 

[nashathedog]

Fair enough, Thanks...

 

[Mauller]

Accident double post.

 

[Mauller]

This is aimed at the HPC and high end server brigade, mate, rather than gaming PCs. Given every test I've seen has shown minimal losses when Crossfiring between PCI-E2 16x and 8x, I think it's going to be a good few generations before PCI-E3 connections are the limiting factor is a games machine.

In the short term, pretty sure this article is only of direct interest to Kaapstad with some of his setups!

But you are comparine PCIE to PCIE, if this has latency improvements then I will have a performance advantage, even on the desktop. Plus we are talking about next gen cards where PCIE3 could quite well be limiting.

But the major point of latency is important when it comes to VR performance and smooth frame delivery in multi GPU setups. Also, PCIE limits multi card when you go beyond a dual card setup. And with DX12 that will show even more.

It also has another advantage on the desktop that I mentioned. Coherent fabric uses fewer data lanes for greater throuput. Meaning a multi GPU setup wont eat the majority of your data lanes from your cpu. Giving you a nice amount left for things like M.2 SSD's etc.

 

[andybird123]

But you are comparine PCIE to PCIE, if this has latency improvements then I will have a performance advantage, even on the desktop. Plus we are talking about next gen cards where PCIE3 could quite well be limiting.

But the major point of latency is important when it comes to VR performance and smooth frame delivery in multi GPU setups. Also, PCIE limits multi card when you go beyond a dual card setup. And with DX12 that will show even more.

It also has another advantage on the desktop that I mentioned. Coherent fabric uses fewer data lanes for greater throuput. Meaning a multi GPU setup wont eat the majority of your data lanes from your cpu. Giving you a nice amount left for things like M.2 SSD's etc.

Yeah, except in those articles AMD are talking about HPC solutions, have they said anywhere they are planning to release this for consumer?

PCI SIG have released the specs for PCIE4, which is up to 128GB/s so that is marginally faster and more likely to be taken up by intel and NV
this seems like a pretty short term stop gap even if they did try to bring it to desktop

 

[humbug]

Yeah, except in those articles AMD are talking about HPC solutions, have they said anywhere they are planning to release this for consumer?

PCI SIG have released the specs for PCIE4, which is up to 128GB/s so that is marginally faster and more likely to be taken up by intel and NV
this seems like a pretty short term stop gap even if they did try to bring it to desktop

2017~ and 64GB/s.

http://www.kitguru.net/components/g...es-and-new-connector-to-be-finalized-by-2017/

 

[Mauller]

Yeah, except in those articles AMD are talking about HPC solutions, have they said anywhere they are planning to release this for consumer?

PCI SIG have released the specs for PCIE4, which is up to 128GB/s so that is marginally faster and more likely to be taken up by intel and NV
this seems like a pretty short term stop gap even if they did try to bring it to desktop

It is 64GB's over 16 lanes as Humbug mentioned. But as i stated, AMD have presented it before that it will be 100GB's over 4 lanes. which is considerably faster per lane. Leading to the situation i mentioned further up.

So overall fewer lanes are needed from a CPU so more performance to more peripherals in the end.

Yeah, they have not mentioned it in consumer space. but they have mentioned it working over standard PCI links and that the data channels will negotiate as to whether a device supports the newer link tech or PCI.

It essentially means that 16 lanes could feed 4 GPU's with 100GB's each. while leaving the other 25+ lanes free for other peripherals.

 

[andybird123]

Do you have any links?
All I can find is its GMI links and it needs to be an MCM package (so not standard desktop stuff)

 

[Mauller]

This was one of them at least. http://fudzilla.com/news/processors/38381-amd-s-new-interconnect-tech-is-coherent-fabric

This is the key point

AMD's GMI - Global Memory Interconnect, extends the coherent fabric between chips on the same multi-chip module (MCM) package. The coherent fabric extends between packages using combo phys which also support PCIe interconnection.

 

[andybird123]

I've read that article, I don't think it says what you are implying - that coherent fabric can just use any four lane PCIE to enable 100gb/s leaving the rest for standard pcie - it is talking purely about a cpu/gpu module, not a CPU with a pcie gpu like a desktop

 

[Mauller]

It mentions chips on the same packages and between package links. plus they have talked about using it between GPU's for server situations. so it implies that it is also an interlink outside of MCM modules.

But we just have to see how it all play out.

 

[humbug]

100GB/s over 4 lanes is a whole lot better than 64GB/s over 16, in some cases perhaps even better than 200GB/s over 16.

PCIe lanes are a limiting factor, if you can push more bandwidth over fewer lanes it matters because you have more lanes available for more of the same and / other PCIe lane IO.

 

[bru]

Well thank god for PCIe 4.0 and hopefully 5.0 following that. If NVidia and now it would seem AMD had their way, with NVlink and now this coherent fabric, first starting off in the server and HPC market, just how long until they decide it is needed on the desktop side of things, 2 or 3 years and then we could have well this motherboard has AMD's super fast system on it whereas this one has NVidia's.

It is bad enough having certain games perform much better or worse on each sides hardware, now with the whole variable refresh rate thing you have to chose a side before you buy a monitor. How long before it extends to motherboards and then what would be next..?

Before anyone starts the whole yes but this one uses open standards yada yada yada.... We all know (or we should if we where honest with our selves) that that is meaningless when you only have two players in the market and neither will use what the other side develops regardless.

 

[Mauller]

Bru, the amd one runs over standard pci link apparently. and the GMI controller will use PCI to communicate or Coherent fabric if the peripherals GMI supports it.

so you could say that it is in extra to PCI rather than a straight up replacement. but saying that, PCI 4 devices will not work on PCI 3 motherboards, they are pin incompatible. although PCI 3 devices can work in a PCI 4 slot.

But from what is stated, Coherent Fabric and Nvlink are faster than PCIe3