8 GPU system possible running from home power socket?

Jules123

Jules123

Jules123

Associate
Joined
24 Nov 2011
Posts
26
Hi,

I have a need to put together a research compute machine consisting of 8 NVidia Titan cards that will be running CUDA. All this running from normal house power socket(s). If it can play games off a multi-monitor set up too, that be good.

Budgets are relatively tight so don't want an expensive enterprise solution. Water cooled be good, can imagine it's going to generate some heat though, not to mention electricity bills.

All the GPU cards must be running off one host computer. So I'm looking for a solution that is more prosumer and I don't mind a bit of DIY.

Thanks for any help,
Jules
 
If you run the rig off of 2 PSUs (2x Enermax Platimax 1500w for example) then you can split the load to 2x13A plugs off of the wall sockets.
I can't foresee you having any problems, other than finding a motherboard that will accept 8 separate cards.
 
The Big Bang Marshal is the mother of all P67 boards. It features eight x16 sized PCI Express slots (4 true, 4 operating at x8 bandwidth)

if you can find it that is, not that many was made

or

EVGA 170-BL-E762-RX XL ATX
which has 7 slots
 
Last edited:
I'd take a look at bitcoin mining rigs, they use similar systems to get as much possible performance out of graphics cards. It won't be pretty, take a look at this google image search and you will see what I mean...

You will need 2 PSUs for that kind of rig, as Rossi said, 2 Enermax Platimax PSUs will be the minimum to be honest!

Either way, you are looking at a killer rig...

YOUR BASKET
8 x Asus GeForce GTX Titan 6144MB GDDR5 PCI-Express Graphics Card £999.95 (£7,999.60)
1 x EVGA SR-X Dual Socket (Socket 2011) Motherboard £559.99
2 x EVGA SuperNova NEX 1500w Digital Classified Modular Power Supply £369.98 (£739.96)
2 x Intel Xeon E5-2620W 2.00GHz (Socket 2011) - Retail £349.99 (£699.98)
2 x Kingston HyperX Beast 16GB (4x4GB) PC3-12800C9 1600MHz Quad Channel Kit (KHX16C9T3K4/16X) £95.99 (£191.98)
Total : £10,207.70 (includes shipping : £13.50).



This doesn't include mounting, SSD(s), hard drives, cooling... and i'd put some higher end RAM in it. There might be cheaper bespoke solutions out there for you...
 
Last edited:
8 Titans and budget is relatively tight?

Ok that aside few issues, the titan is a dual slot card, so finding a case with all those slots. . . I have no idea if one even exists. Ok so water cooling, is there a single slot water block available? Next issue I'm not sure you'll find a none enterprise solution chipset with enough bandwidth available for 8 titans.

To be honest I think you would be better looking at a NVidia telsa gpu node. Someone like Super Micro makes a 8 gpu node setup.
 
This might serve you better, at less than half the price and fully contained within the case:

YOUR BASKET
1 x Asus Z9PE-D8 WS Dual Socket C602 Chipset (Socket 2011) DDR3 Motherboard £459.95
7 x PNY Nvidia Quadro 2000 Graphics Card - 1GB - GDDR5 SDRAM £419.99 (£2,939.93)
1 x EVGA SuperNova NEX 1500w Digital Classified Modular Power Supply £369.98
2 x Intel Xeon E5-2620W 2.00GHz (Socket 2011) - Retail £349.99 (£699.98)
1 x Lian Li PC-A71FB Aluminium Full Tower Case - Black £199.99
1 x Intel 335 Series 20nm 240GB 2.5" SATA 6Gb/s Solid State Hard Drive - Retail (SSDSC2CT240A4K5) £169.99
2 x Kingston HyperX Genesis 16GB (4x4GB) DDR3 PC3-12800C9 1600MHz Dual/Quad Channel Kit (KHX1600C9D3K4/16GX) £99.95 (£199.90)
2 x Alpenföhn Groß Clockner Rev. B CPU Cooler (Socket 775 / 1155 / 1156 / 1366 / AM2 / AM2+ / AM3 / FM1 / FM2) £29.99 (£59.98)
Total : £5,117.39 (includes shipping : £14.75).



One downside - I don't know if the graphics cards can work in 8x slots, or if this many can work together. But if you know how to get 8 titans working together, it is probably reasonably simple to do the same thing on the Quadro cards.
 
Rooster212 said:
I'd take a look at bitcoin mining rigs, they use similar systems to get as much possible performance out of graphics cards. It won't be pretty, take a look at this google image search and you will see what I mean...

You will need 2 PSUs for that kind of rig, as Rossi said, 2 Enermax Platimax PSUs will be the minimum to be honest!

Either way, you are looking at a killer rig...

YOUR BASKET
8 x Asus GeForce GTX Titan 6144MB GDDR5 PCI-Express Graphics Card £999.95 (£7,999.60)
1 x EVGA SR-X Dual Socket (Socket 2011) Motherboard £559.99
2 x EVGA SuperNova NEX 1500w Digital Classified Modular Power Supply £369.98 (£739.96)
2 x Intel Xeon E5-2620W 2.00GHz (Socket 2011) - Retail £349.99 (£699.98)
2 x Kingston HyperX Beast 16GB (4x4GB) PC3-12800C9 1600MHz Quad Channel Kit (KHX16C9T3K4/16X) £95.99 (£191.98)
Total : £10,207.70 (includes shipping : £13.50).



This doesn't include mounting, SSD(s), hard drives, cooling... and i'd put some higher end RAM in it. There might be cheaper bespoke solutions out there for you...
Click to expand...

Mobo only has 7 PCI-e Slots.
 
Yes it will work off normal socket. I currently have one set up at work. Its not cheap mind you the chassis/mobo/psus are about £5K + VAT. Its also no good with playing games as the cards do not work in SLI. ITs a GPU Compute unit that is specifically catered to a job. I am not even sure if it supports windows os as this one is being installed with Ubuntu. Oh and on top of that.. I hope you live in a detached house (Preferably in the middle of nowhere) As when its fully populated and running flat out it sounds like a 747. The fans are about 2" thick and are paired up.. :eek:
 
ZXSpectrum said:
I hope you live in a detached house (Preferably in the middle of nowhere) As when its fully populated and running flat out it sounds like a 747. The fans are about 2" thick and are paired up.. :eek:
Click to expand...
Everything goes up in the attic conversion, no problem up there. Maybe keep the birds awake though?

I also noticed that solutions can't fit eight cards, Titans are dual slot, or am I missing something?

The stuff I am doing will be compute intensive so I don't think older cards are going to cut it. I'll be also using dynamic parallelism in CUDA supported only on GK110s.

How does a 16x to dual 8x riser work?

Cheers,
Jules
 
Does it really have to be a single computer? Clusters tend to make more sense for heavy number crunching than single machines. You lose bandwidth between nodes relative to pci-e across a single motherboard, but you get more nodes and less risk of catastrophic hardware failure. Two computers with four cards in each will be logistically a lot easier than one computer with eight cards - and four computers with two cards in each may be better still.
 
Jules123 said:
Everything goes up in the attic conversion, no problem up there. Maybe keep the birds awake though?

I also noticed that solutions can't fit eight cards, Titans are dual slot, or am I missing something?

The stuff I am doing will be compute intensive so I don't think older cards are going to cut it. I'll be also using dynamic parallelism in CUDA supported only on GK110s.

How does a 16x to dual 8x riser work?

Cheers,
Jules
Click to expand...

The PCIe slots are spaced in 2 slot intervals so you can fit 8 cards. The one that is currently in test has 8xTesla K20's in it. I have also tried it with 8 x GTX 680. I have not used the 16x to dual 8x Riser.

Also forgot to mention. If you do fit Geforce/GTX/Titan cards then you will need to modify the internal cowling/ducting as it is designed for Tesla cards and they have the power on the end of the card rather than the top like the GTX cards..
 
Last edited:
Jules123 said:
I also noticed that solutions can't fit eight cards, Titans are dual slot, or am I missing something?
Click to expand...

Rooster212 said:
I'd take a look at bitcoin mining rigs, they use similar systems to get as much possible performance out of graphics cards. It won't be pretty, take a look at this google image search and you will see what I mean...
Click to expand...

You'll have to do something like that unless you can find a case with 16 expansion slots.
 
ZXSpectrum said:
The PCIe slots are spaced in 2 slot intervals so you can fit 8 cards. The one that is currently in test has 8xTesla K20's in it. I have also tried it with 8 x GTX 680.
Click to expand...
Okay, I get it, 2 slot spacing.

I didn;t think it would be possible to do all this on one computer. Can you elaborate more on the cooling? Do you know what temperatures you were getting? Did you OC the cards?

Cheers,
Jules
 
You must log in or register to reply here.
Back
Top Bottom