One thing nVidia does do well is make good looking hardware - photos don't do the FE cards justice and some of the server and compute systems look really nice like the DGX, etc.
Took the plunge - First Home Server
- Thread starter BogEyes
- Start date
One thing nVidia does do well is make good looking hardware - photos don't do the FE cards justice and some of the server and compute systems look really nice like the DGX, etc.
- Joined
- 7 Jan 2006
- Posts
- 1,566
- Location
- somewhere between venus and mars
Changing the tune. Thanks everyone has read and responded, warts and all. Everyone has encouraged me to go and read things I didn’t know existed last week.
I think part of why I might have rustled a few feathers is that I’ve just done a poor job of separating what I want to learn long term, from what I expect to achieve immediately.
To reset expectations a bit for anyone who is still interested:
Short Term.
- Accumulate enough hardware to start designing an enclosure
- Start a build thread
- Source one OEM heatsink, reverse engineer how it mounts to the V100, and build some prototypes
- Learn fluid and thermal dynamics to an A-Level understanding, and then put a wanted ad up in the members market for old heatsinks
- 3D Print a better testing bench for my worktop
- Check the Motherboard, CPU and Ram posts and is stable with my 4070
Mid term.
- 2× V100's
- 1x Xeon
- 4x DIMMs
- No attempts at VRAM pooling, NVLink, or scaling beyond what the board supports by default.
- Get a stable platform I can benchmark, stress, cool, and understand.
Longer term.
- Completely wide open, other than just trying to learn more about computers and how they work and what they can do beyond play games.
To make it clear, I fully accept;
- V100s, Xeons and DDR4 are legacy hardware in AI terms
- Support/drivers/new models will lock me out sooner, rather than later
-This will never compete with hosted and subscription based models
Where I would appreciate advice and help:
Known HBAs that will be suitable for Unraid with SAS-12Gbps drives
Practical cooling approaches from people who have used enterprise gpus outside of a rack chassis
Any known quirks with the Asus Z10PE-D8 WS, maybe something I should watch out for that isn't obvious in the manual.
BIOS settings worth changing (or not changing) for initial system stability
I mean this in as genuine a way as I can possibly convey; This isn’t about trying to build something that is going to be better than anything I can pay for, it won't be. It’s about understanding hardware, software, and an entire area of computers I have no previous experience of. Rather than moping about because I feel that i've missed the boat with regards to ever having a career in engineering, design or creativity. I'm just going to have a go at home, and see how I get on learning something new that interests me.
If the bloody thing ends up being loud, power hungry and just too old to do anything useful with, so be it – but I’d rather learn that from building it and it failing, than not just trying at all and going to the pub.
And for anyone who is still reading, I will start a build thread once parts are 'on the bench'
I think part of why I might have rustled a few feathers is that I’ve just done a poor job of separating what I want to learn long term, from what I expect to achieve immediately.
To reset expectations a bit for anyone who is still interested:
Short Term.
- Accumulate enough hardware to start designing an enclosure
- Start a build thread
- Source one OEM heatsink, reverse engineer how it mounts to the V100, and build some prototypes
- Learn fluid and thermal dynamics to an A-Level understanding, and then put a wanted ad up in the members market for old heatsinks
- 3D Print a better testing bench for my worktop
- Check the Motherboard, CPU and Ram posts and is stable with my 4070
Mid term.
- 2× V100's
- 1x Xeon
- 4x DIMMs
- No attempts at VRAM pooling, NVLink, or scaling beyond what the board supports by default.
- Get a stable platform I can benchmark, stress, cool, and understand.
Longer term.
- Completely wide open, other than just trying to learn more about computers and how they work and what they can do beyond play games.
To make it clear, I fully accept;
- V100s, Xeons and DDR4 are legacy hardware in AI terms
- Support/drivers/new models will lock me out sooner, rather than later
-This will never compete with hosted and subscription based models
Where I would appreciate advice and help:
Known HBAs that will be suitable for Unraid with SAS-12Gbps drives
Practical cooling approaches from people who have used enterprise gpus outside of a rack chassis
Any known quirks with the Asus Z10PE-D8 WS, maybe something I should watch out for that isn't obvious in the manual.
BIOS settings worth changing (or not changing) for initial system stability
I mean this in as genuine a way as I can possibly convey; This isn’t about trying to build something that is going to be better than anything I can pay for, it won't be. It’s about understanding hardware, software, and an entire area of computers I have no previous experience of. Rather than moping about because I feel that i've missed the boat with regards to ever having a career in engineering, design or creativity. I'm just going to have a go at home, and see how I get on learning something new that interests me.
If the bloody thing ends up being loud, power hungry and just too old to do anything useful with, so be it – but I’d rather learn that from building it and it failing, than not just trying at all and going to the pub.
And for anyone who is still reading, I will start a build thread once parts are 'on the bench'
HBAs:
With HBAs you want to make sure they are preflashed/configured in IT mode.. People advertise them as compatible with TrueNAS/Unraid..
e.,g.https://www.ebay.co.uk/itm/404487396328
Server GPUs
Simple, they are designed to be cooled from directed airflow mid chassis, you can buy those in various types, either blower or fans on a shroud cheaply, albeit, bloody noisy with fans at full tilt.
www.aliexpress.com
You can buy waterblocks for water cooling the V100, so you could do that, but they aren't cheap (~£140 each), however, the system would be very cool/quiet.. You can pick up cheap watercooling stuff on the for sale section here, it's where I get a lot of stuff.
With HBAs you want to make sure they are preflashed/configured in IT mode.. People advertise them as compatible with TrueNAS/Unraid..
e.,g.https://www.ebay.co.uk/itm/404487396328
Server GPUs
Simple, they are designed to be cooled from directed airflow mid chassis, you can buy those in various types, either blower or fans on a shroud cheaply, albeit, bloody noisy with fans at full tilt.
P40 M40 P100 V100 GPU Cooler Dual Turbo Fan Universal Ultra Thin Heat Sink for Graphics Card - AliExpress
Smarter Shopping, Better Living! Aliexpress.com
www.aliexpress.com
You can buy waterblocks for water cooling the V100, so you could do that, but they aren't cheap (~£140 each), however, the system would be very cool/quiet.. You can pick up cheap watercooling stuff on the for sale section here, it's where I get a lot of stuff.
Last edited:
Changing the tune. Thanks everyone has read and responded, warts and all. Everyone has encouraged me to go and read things I didn’t know existed last week.
I think part of why I might have rustled a few feathers is that I’ve just done a poor job of separating what I want to learn long term, from what I expect to achieve immediately.
To reset expectations a bit for anyone who is still interested:
Short Term.
- Accumulate enough hardware to start designing an enclosure
- Start a build thread
- Source one OEM heatsink, reverse engineer how it mounts to the V100, and build some prototypes
- Learn fluid and thermal dynamics to an A-Level understanding, and then put a wanted ad up in the members market for old heatsinks
- 3D Print a better testing bench for my worktop
- Check the Motherboard, CPU and Ram posts and is stable with my 4070
Mid term.
- 2× V100's
- 1x Xeon
- 4x DIMMs
- No attempts at VRAM pooling, NVLink, or scaling beyond what the board supports by default.
- Get a stable platform I can benchmark, stress, cool, and understand.
Longer term.
- Completely wide open, other than just trying to learn more about computers and how they work and what they can do beyond play games.
To make it clear, I fully accept;
- V100s, Xeons and DDR4 are legacy hardware in AI terms
- Support/drivers/new models will lock me out sooner, rather than later
-This will never compete with hosted and subscription based models
Where I would appreciate advice and help:
Known HBAs that will be suitable for Unraid with SAS-12Gbps drives
Practical cooling approaches from people who have used enterprise gpus outside of a rack chassis
Any known quirks with the Asus Z10PE-D8 WS, maybe something I should watch out for that isn't obvious in the manual.
BIOS settings worth changing (or not changing) for initial system stability
I mean this in as genuine a way as I can possibly convey; This isn’t about trying to build something that is going to be better than anything I can pay for, it won't be. It’s about understanding hardware, software, and an entire area of computers I have no previous experience of. Rather than moping about because I feel that i've missed the boat with regards to ever having a career in engineering, design or creativity. I'm just going to have a go at home, and see how I get on learning something new that interests me.
If the bloody thing ends up being loud, power hungry and just too old to do anything useful with, so be it – but I’d rather learn that from building it and it failing, than not just trying at all and going to the pub.
And for anyone who is still reading, I will start a build thread once parts are 'on the bench'
You need to be very careful with SMX chip as it’s a very delicate installation process. If you stick to a reasonable config you should get a usable system. By that, I mean something like 4 VMs with a GPU each. Once you start to share resources across the platform is where you’ll hit roadblocks and expose the technology flaws in the platform. It would be a great learning experience but, I’d avoid that as its a pretty futile task and for AI you efforts would certainly be much better spent working with ROCm and OneAPI.
Ps ignore much of what Rroff says, he’s one of those Google expert types with no experience.
Last edited:
Its not an entirely straighforward answer I'm afraid
The memory is quad-channel LPDDR5-8000 so effectively that gives a theoretical memory bandwidth of 4 x 64GB = 256GB/sec.
However its unified memory and hence dependent on the memory connections to the infinity fabric and the MALL (infinity cache). These differ between CPU and graphics on Strix Halo.
The CPU is essentially a 9950X (dual-CCD) without 32MB MALL (infinity cache) so its actually a little slower than the desktop 9950X CPU. On Strix Halo the 8060S compute units have the 32MB MALL dedicated to them, more on that later.
As a consequence the CPU on Strix Halo is limited to 32 bytes/cycle for writing to memory so its a bit slower writing than reading. Just like desktop CPUs each CCD is effectively limited to 64GB/sec bandwidth so the upshot of all of this is the CPU (both CCDs) will max out on read/modify/add at around 175GB/sec. 120GB/sec read, 80GB/sec write.
The 8060S graphics/compute effectively gets the full bandwidth and all the MALL (infinity cache). That means in practice you have around 220-240GB/sec bandwidth for the compute units although its very dependent on workload - anything involving the CPU (other than scripts) will pull the write bandwidth down. memtest_vulkan is about 220/198GB/sec. It's quite amusing watching some AI workloads on Strix Halo - CPU running at 1.45GHz (idling) while the GPU is at 2.9GHz with compute at 98% and 100+GB memory in use.
There is also the XDNA2 NPU attached to the infinity fabric which I think has 64GB/sec read/write bandwidth although that's a guess. Its pretty hard to find a way to check as its undocumented AFAIK.
There's some better explanations over at chipsandcheese :
Evaluating the Infinity Cache in AMD Strix Halo
Strix Halo is the codename for AMD’s highest end mobile chip, which is used in the Ryzen AI MAX series.
chipsandcheese.com
Basically its a 4.5L box - similar size to the DGX Spark (quelle surprise!) - which pulls a max of about 140W and is just about on par with the DGX Spark for inferencing, not so much with diffusion but its getting there with ROCm. Two Gen4 SSD slots and varying I/O depending on manufacturer. Plays games at around 4060M/4070M/PS5 standard - ie fine up to 1440p.
Edit - in terms of AI model sizes and memory use, it depends. For inferencing something like gpt-oss:120b will use 64GB of compute (graphics) memory and another 14/15GB of CPU memory is in use. 78GB committed. Diffusion is very variable depending on model/image etc but I regularly have over 100GB committed and on one occasion 122GB which wasn't great in terms of speed.
Edit2 - since there's a load of Strix Halo stuff here anyway I thought I'd link this chipsandcheese video for anyone techie enough as newer ROCm does the s/w stuff he talks about regarding memory/MALL allocation between CU and NPU :
Last edited:
This is the problem for me. You just shouldn't use these because not only are they not going to do what you hope, they're going to cost you more money.
The rest I can quite understand as I did much the same myself back in the 00's - HP 1U DL-145 , dual opterons, howled like a banshee. I built a box for it in the garage, attached all sorts to it and retired it 8 years later for a box that did everything the old box did and used a quarter of the power to do it. Right enough mine wasn't 10 years out of date off the bat
On anything related to AI, last year is last decade so to speak.
I don't know how much of the NVLINK discussion is relevant.
OP said they are using SMX to PCI-E adaptors and I'm not aware these cheap adaptors have any NVLINK connectivity since NVLINK is a specialist high bandwidth bus.
You can buy sub boards designed to hold multiple SMX GPU with integrated NVLINK support which then connect via PCI-E but these are rare, expensive and many are vendor locked.
As I read it, OP will connect these cards over PCI-E.
Multi GPU over PCI-E is usable for inference, it takes around 1-2 minuted to load a 70b 4.5bpm Llama into 48GB with some context but that is a one time effect.
PCI-E bandwidth doesn't make a lot of difference during inference, only the context is sent to each card and 2 cards should spit out 10/15 Tokens a second with a 20-30Gb model.
That's about reading speed for me so is usable for chat responses.
Jump up to 70b and 3 cards and more like 6-8 tokens a second which is slow but for simple chat bot interactions.
Not so good for coding etc. as you are limited how much 'code' you can upload in the small context and if you want 20k tokens of code back, a small project... that will be around half an hour to an hour depending on the model and it gets worse the large the model and the more cards you add.
PCI-E is no real use for training, it's just to way slow for anything that doesn't fit on one card, but then you can just rent some cloud compute.
If you are going to play with the SMX, I'd still look at proper coolers. From the popular China site, SMX HSF's are around £50 and you need to add high pressure fans... not powered of the motherboard fan header. Water blocks are £35 but you need pump and radiator and radiator fans. The GPU is a precision part, unless you have CAD drawings and a high precision mill that can work suitable materials you aren't going to make anything better for the cost. Those are bare cores, uneven pressure, you crack it, not enough pressure, hot spots, partly not covered, it burns up. The die itself is 815mm^2 compared to a 5090 at 750mm^2 so it's huge. You'll pay close to that for a bare cooler of something else that doesn't fit and has no chance of effectively cooling it. Even most CPU tower coolers with heat pipes will likely be inadequate at 300W, assuming you have some way of properly cooling all the voltage regulations too.
OP said they are using SMX to PCI-E adaptors and I'm not aware these cheap adaptors have any NVLINK connectivity since NVLINK is a specialist high bandwidth bus.
You can buy sub boards designed to hold multiple SMX GPU with integrated NVLINK support which then connect via PCI-E but these are rare, expensive and many are vendor locked.
As I read it, OP will connect these cards over PCI-E.
Multi GPU over PCI-E is usable for inference, it takes around 1-2 minuted to load a 70b 4.5bpm Llama into 48GB with some context but that is a one time effect.
PCI-E bandwidth doesn't make a lot of difference during inference, only the context is sent to each card and 2 cards should spit out 10/15 Tokens a second with a 20-30Gb model.
That's about reading speed for me so is usable for chat responses.
Jump up to 70b and 3 cards and more like 6-8 tokens a second which is slow but for simple chat bot interactions.
Not so good for coding etc. as you are limited how much 'code' you can upload in the small context and if you want 20k tokens of code back, a small project... that will be around half an hour to an hour depending on the model and it gets worse the large the model and the more cards you add.
PCI-E is no real use for training, it's just to way slow for anything that doesn't fit on one card, but then you can just rent some cloud compute.
If you are going to play with the SMX, I'd still look at proper coolers. From the popular China site, SMX HSF's are around £50 and you need to add high pressure fans... not powered of the motherboard fan header. Water blocks are £35 but you need pump and radiator and radiator fans. The GPU is a precision part, unless you have CAD drawings and a high precision mill that can work suitable materials you aren't going to make anything better for the cost. Those are bare cores, uneven pressure, you crack it, not enough pressure, hot spots, partly not covered, it burns up. The die itself is 815mm^2 compared to a 5090 at 750mm^2 so it's huge. You'll pay close to that for a bare cooler of something else that doesn't fit and has no chance of effectively cooling it. Even most CPU tower coolers with heat pipes will likely be inadequate at 300W, assuming you have some way of properly cooling all the voltage regulations too.
There is no NVLink functionality using those adapters. The main thing I was trying to clear up is that the DMI bus, and the inherent limitations which would if it was the case make them cripplingly slow, is not relevant to using multiples of these cards in this context contrary to the claim of one poster. As you said there are a lot of considerations to using the cards like this for things like LLMs which needs working with.
There are some "NVLink Lite" multi card boards available fairly cheap these days which AFAIK aren't vendor locked but that is beyond my experience. (I might be wrong but I don't believe they support any kind of automatic resource pooling and an LLM would need to be programmed to take advantage of the benefits of NVLink for them to provide a substantial benefit).
PS it is SXM not that it is a big deal but I notice people keep saying SMX in this thread.
Last edited:
I hope we didn't overcook you
As an aside in regard to the delicacy of the SXM mounting process it is only relevant to the heat sink really unless you go at it ham fisted and while there is a certain amount of art/experience to knowing when to stop if you use finger grip rather than palm most people can't exceed the maximum torque required to damage the die (palm grip can exceed the maximum torque by around 3x).
- Joined
- 7 Jan 2006
- Posts
- 1,566
- Location
- somewhere between venus and mars
Been a bit sidetracked and doing a lot more research off the back of many of the points you've all made.
@Demon thanks for the HBA recommendation. Re: the Ali blower fan setup, without a GPU shroud I'm not sure it would work. However, I do like the idea of going the watercooled route now. After getting some more accurate measurements of the V100 die, I think it might be the best option.
@Vestas the Strix Holo and DGX Spark are interesting no doubt. It seems they could do everything I need and more. However, for me, buying something ready made that just works, although sensible, doesn't appeal to my inner 'i'll build one in the shed instead' mentality. If I needed to get up and running tomorrow, this would definitely be the route to take, so thanks for all the extra info and linked vids.
re: 2x V100's
You're right, you're right. 2x 16GB V100's will not do what I hope...*cue evil laugh*
@decto
After your post, I've decided against the PCI-E route. There are too many compromises, and although I enjoy a good 'hack', I think that unless I'm utlising NVlink to get the full bandwidth available to the V100's, there is no point to the project at all. It was your comment regarding training that was the spanner in the works for me, that has made me re-think my entire plan, so thank you Those Ali waterblocks don't look like they cool the VRM's to me, also I've never watercooled any PC parts before. However, I do think it's going to be the way forward.
@Rroff
It was those adaptors I had been looking at. But between yourself and decto, i've gone off the idea completely. Re: The SXM2 mounting, yeah it's only because I have calibrated torque wrenches that I feel comfortable going down this route.
oh, and @Vestas
Not at all, I think you all have helped me see the holes in my original plan, and have sent me off down a more small engineering project pathway, rather than a, lets just buy all these different odds and sods and hope it will work.
My new plan it to just see if I can fabricate a waterblock for one v100 as a small CNC project and, everything else is on hold until then.
@Demon thanks for the HBA recommendation. Re: the Ali blower fan setup, without a GPU shroud I'm not sure it would work. However, I do like the idea of going the watercooled route now. After getting some more accurate measurements of the V100 die, I think it might be the best option.
@Vestas the Strix Holo and DGX Spark are interesting no doubt. It seems they could do everything I need and more. However, for me, buying something ready made that just works, although sensible, doesn't appeal to my inner 'i'll build one in the shed instead' mentality. If I needed to get up and running tomorrow, this would definitely be the route to take, so thanks for all the extra info and linked vids.
re: 2x V100's
You're right, you're right. 2x 16GB V100's will not do what I hope...*cue evil laugh*
@decto
After your post, I've decided against the PCI-E route. There are too many compromises, and although I enjoy a good 'hack', I think that unless I'm utlising NVlink to get the full bandwidth available to the V100's, there is no point to the project at all. It was your comment regarding training that was the spanner in the works for me, that has made me re-think my entire plan, so thank you
Those Ali waterblocks don't look like they cool the VRM's to me, also I've never watercooled any PC parts before. However, I do think it's going to be the way forward.@Rroff
It was those adaptors I had been looking at. But between yourself and decto, i've gone off the idea completely. Re: The SXM2 mounting, yeah it's only because I have calibrated torque wrenches that I feel comfortable going down this route.
oh, and @Vestas
I hope we didn't overcook you
Not at all, I think you all have helped me see the holes in my original plan, and have sent me off down a more small engineering project pathway, rather than a, lets just buy all these different odds and sods and hope it will work.
My new plan it to just see if I can fabricate a waterblock for one v100 as a small CNC project and, everything else is on hold until then.
Should have known given your background you mentioned hah.
Good man
For the required surface area you really need to skive the internal fins for a waterblock or it could end up quite inefficient.
If it was me (I watercool my PCs) I'd look at the slightly simpler route, and create a modern equivalent of the NZXT G12 which was a generic bracket that allowed a fairly common/normal AIO (All in one) CPU Cooler be repurposed to mount directly to the die of the GPU, and provided a built in fan for memory/vrm cooling..
*** No competitor linking ***
These are no longer sold, I did have one a while back and it was a fairly good hybrid solution, and since you can reuse cheaper AIO coolers, the cost/convenience is perfect.. obviously with your CNC skills you would just need to make a similar bracket rather than the full waterblock..
Last edited by a moderator:
You really need a torque screwdriver accurate to 1/10th of a Nm. I’d read the official installation documentation and anything that comes with the carrier card TBH as you’re painting pretty far out of the lines and the carrier card is a big unknown. I’d definitely avoid using riser cables if you do end up machining your own water blocks. The trick will be keeping everything stable and the tubing accurately aligned because it wouldn’t take much twisting force to change the amount of torque on the whole stack.
- Joined
- 7 Jan 2006
- Posts
- 1,566
- Location
- somewhere between venus and mars
Yeah, I'm toying around with a couple of different mounting solutions using the 3d printer atm.
I do think water cooling is the right direction to go, as although many CPU heatsinks (Air) on paper should be able to cool a power limited V100, all of that leverage un settles me.
I've never watercooled until using an AIO CPU cooler recently, and I don't think that really counts being a closed system. I'm trying to fully wrap my head around how waterblocks and AIO's work.
Also, if anyone has any idea on what temp the VRM stacks get to on the V100 that would be really useful information.
I do think water cooling is the right direction to go, as although many CPU heatsinks (Air) on paper should be able to cool a power limited V100, all of that leverage un settles me.
I've never watercooled until using an AIO CPU cooler recently, and I don't think that really counts being a closed system. I'm trying to fully wrap my head around how waterblocks and AIO's work.
Also, if anyone has any idea on what temp the VRM stacks get to on the V100 that would be really useful information.
You really need to read the Nvidia documentation and understand the rigidity of the carrier board. The VRM temps will depend on the cooling and to understand that you need to do the TDP calculations.
Turning up a water block to deal with 300 watt load will likely take a few attempts and a fair bit of testing. Once you’ve done that the other side of the heat exchange is pretty straightforward. Decent pump, 3/8 - 1/2 tubing, 120.3/4 rad and fans.
Personally I’d focus on first building and testing with the official HSF to confirm you have a functional system. I’d then move into the getting everything configured and running the intended workloads.
Very old CPUs for this day and age for llm use.
If you had the ddr4 single socket 5950x would beat that setup easily but if I wanted something for work like this I'd want zen 5 at the least. Would take 8 zen 5 cores over 16 haswell cores any day.
- Joined
- 7 Jan 2006
- Posts
- 1,566
- Location
- somewhere between venus and mars
@Cyber-Mav
I would like to utilise and experiment with NV-Link rather than depending upon PCI-E or Unified memory for LLM learning purposes.
What are you (and @Armageus), suggesting exactly?
I would like to utilise and experiment with NV-Link rather than depending upon PCI-E or Unified memory for LLM learning purposes.
What are you (and @Armageus), suggesting exactly?
Last edited: