Project Gemini Rebuild (SR-2)

[Biffa]

So, like many of my builds this one has a history

It has been a AMD XP1800, E6300, Q6600 B3, then a Q6600 G0 @3.4Ghz then lastly an i7 920 D0 @ 4.0Ghz

Cooling wise its been on air, H20-I20, H50, and then Custom water.

It was the first rig I ever watercooled, and doubles up as my VMWare server, running 3 hosts.

It also is a 24/7 Folding@home machine and at some state (probably at the E6300 stage) got named Gemini

Lets concentrate on the most recent spec, then go onto the build spec and progress from there

Pre-Upgrade spec:

Cpu: i7 920 @ 4.2Ghz
Mobo: Asus P6T X58
Ram: OCZ 13333 DDR3 Low Voltage 12GB (6x2GB)
Case: Lian-Li A16B
CPU Block: XSPC V3
Pump: EK 4.0
Rad: XSPC RS240
Res: EK Multires
Tubing/fittings: 8-10mm white tubing on compression fittings.

Threw the build together from spares (bought the pump) after an unsuccesful build with the H50 (too hot) and this is how it looked.

Sitting at around 70°C and pushing somewhere around 33K PPD on a bigadv 24/7 it was stable as a rock.

But I had a plan, I wanted to reduce the number of GPU's I was running in my other machines, yet not lose any output and save some money on the electric bills.

And then the hardware was released that would make Gemini live to its name....

The EVGA Classified SR-2 (Super Record 2)

With dual 1366 sockets for Xeon CPU's and Overclockable to boot the spec sheet read like a hardware nuts wet dream.

EVGA Classified SR-2 (Super Record 2)
Part Number	270-WS-W555-A1
CPU Support	Intel Socket 1366 (Xeon 5600/5500)
Chipset		Intel 5520
Memory Support	Dual or Triple Channel DDR3
SLI Support	2-way, 3-way, 4-way
SATA Header	8
SATA Type	2x600MB/sec / 6x300MB/sec
USB Ports	2x3.0 / 10x2.0
PCIE Slot	7 x PCIe x16/8
Integrated LAN	2 x 10/100/1000

It would be mine.. oh yes!

 

[Biffa]

So.. here we go

Rebuild Spec:

EVGA SR-2 Motherboard
Dual Xeon X5650
Dual EK Extreme Blocks
EK SR-2 Motherboard block
Thermochill PA140.3 Radiator
EK-Reservoir Combo DCP 2.2
Lian-Li PC-P80 Case

Someone obviously approves:

The case was a tricky one, it had to be big enough to take the crazy sized SR-2 Motherboard:

You need a case that can take the depth of the motherboard, currently there are only 2 or 3 stock cases that can do it, or you can get a custom MM mobo tray and hack it into a Corsair. The only other options are to buy a MountainMods case itself or a DangerDen Plexi case. Neither of which rock my world.

The PC-V2010 fits, but thats even more expensive, and not as good for watercooling I reckon.

Lucky the case isnt going to be on display anywhere as its an ugly mofo, this isn't going to be an elegant build, its going to be functional and I didn't want to spend much time modding this baby, I couldn't afford to spend too much time during the transplant of equipment to the new rig stage, as the VMWare images it hosts run some of my stuff that I don't want down for too long a time.

First thing you notice is how big the box for the Lian-Li PC-P80 is, and then you realize the size of the box for the SR-2!

Here it is compared to the box for the Asus P6T7-WS, itself no small potatoes in the Motherboard world and central to another build of mine.

On the flip side I was slightly dissapointed on the box size of the Xeon CPU's.. I mean they were so small I almost didn't notice them at the bottom of the box!
Compared to a standard i7-920 box or the beefy i7-970 box and you can feel short changed considering they cost the same as the 970

At this point I was almost ready to begin, but I was waiting for a few extra parts including the fittings, the res and tubing, so I threw the cpus in, jury rigged a loop with my old faithful XSPC 750 bayres and checked everything booted up.

Then it was time to start putting it all together

 

[Biffa]

Unpacking shots.

The SR-2, how well is this thing packed!

Nekkid in all its glory.. lots of warning stickers everywhere, nomnomnomnomnom

The mobo block:

Nice and shiny, then a trial placement to makes sure I bought the right part, and to see what it would look like

Lets fit that sucker!
Nice easy screws to undo, thats what I like to see.

Nekkid mobo:

My weapon of choice for cleaning computer parts:

Squeaky clean

Fitted.. *phew*

Which reminds me I hate the EK method on all their blocks of using plastic standoffs "stuck" with TIM. Its twice as bad with a block this size but I managed to fit it on the first try without losing a standoff.

 

[Biffa]

That truely is a thing of beauty

Can't wait to see this come together!!

OOOOOO more pics please

Loving it so far!

gt

Thanks guys.. more pics later

 

[Biffa]

ooooh subbed.

id be very interested on how this compares to a single i7. i was saying to someone the other day that id love to do a twin xeon build next but they were insisting that a single i7 clocked can be just as effective?

Depends completely on what you want to do, and whether the application scales more than 4 threads.

So rendering, measured with something like CineBench 10 shows +50% improvement in rendering times.

But games show little or even negative improvement, they show little improvement going from 2 to 4 or 4 to 6 cores so 12/24 isn't going to make any difference

But as a server or distributed computing or scientific computing machine, thats where multi cpus and especially the SR-2 because it can be overclocked to improve results even more

 

[Biffa]

No, this isn't really a show pony, its going to be a functional build at best. I can't help myself with some things as you will see but I don't have a problem with the bare aluminium interior of a case normally anyway.

 

[Biffa]

What Gcard you sticking into it?

THE BEAST THAT IS...

Graphics Bus Technology 	PCI Express
Memory Interface 		64-bit
Memory Bandwidth 		5.3GB/s
Fill Rate 			1.4 Billion pixels/sec
Vertices/Second (Millions) 	263
RAMDACs 			400MHz

 

[Biffa]

I see we have the same taste in cases

I thought Rjk built you one already?

Not much choice really depending on budget, I'm no master modder like yourself so just went for one that fits that didn't require major hacking

i want to see 4.2 out of that rig minimum!

Oh yes, as long as its 24/7 100% stable at 100% on 24 threads

Trying your best to keep Ba at bay ?

Guy can't be top at everything he tries his hand at..

But big respect to that 48core Opteron he has, nothing can compete with that unless they bring out 8 core Westmeres.

 

[Biffa]

Fitting CPU Blocks

CPU Socket and close up of the PWM block.

I think we might be needing these soon! 24 threads of Dual Hex core cpu pr0n!

But first Houston we have a problem.

Xeon motherboard comes with special Xeon backplates designed for Xeon heatsinks. For our waterblocks we need to change them for 1366 ones.
These backplates are held on by the CPU socket surround, which has to be removed to change them. Nicely EVGA include the 1366 parts in the box

Original backplates:

Removing the socket bracket, and Nekkid socket!! You push on those screws and the backplate pops off.

Left one replaced with 1366 compatible backplate:

Fitting the EK Backplate:

Phew.. then rinse and repeat for the other one.

Nice finish on the EK Supreme HF blocks.. also managed a win in getting the newer design. And that puppy is FITTED! (Always remember to remove the protective plastic before bolting down!)

 

[Biffa]

Rads, Fans, Fittings and Fiddling

But first, ime to pause and appreciate the smorgasboard of water cooling goodness with a couple of my better shots

Its a damn tight fit sometimes between "arms" on the Supreme HF and the mobo block, so watch out for that.

Aha.. look what arrived, these might come in handy perhaps. I'm sticking to my favourite fitting/tubing size which is 3\8" ID x 5\8" OD (10-16mm).

Now Rinse that Rad! Till it comes out clean! (dilute distilled vinegar)

Then hot water, and a final rinse with distilled.

Its a PA of course so we need those 3/8" to 1/4" converters.. done! Add the fans and we are ready to find a place to fit it in the case.

 

[Biffa]

Looking awesome. Going to be a folding beast Are the 3 470s going in there?

Oh, and your cat reminds me of the cat in Men in Black!

What's the newer design supreme HF?

No, no GPUs just bigadv. Problem with GPU's is:

Law of diminishing returns and power usage.

The GPU3 code is not very efficient when it comes to sharing CPU interrupts with the CPU, yes you can fiddle with priorities and set affinitys but at the end of the day the return you get from running the GPU's eats away at the performance of the CPU running bigadv and also each other.

e.g. A nicely overclocked sr-2 system at say 4.2-4.4Ghz can pull in 150-160K PPD. Add 4 GPU's in there and that would drop to around 90-100K per day if you use the most powerful GTX480/570 and the GPU's would give you up to say 15-16K each or roughly 60K, maybe equalling the bigadv rig in output.

Plus the power usage skyrockets. A SR-2 pulling 150-160K PPD on its own uses around 700W but add the gpu's in there and you are talking 16-1700W

Now thats high end GPU's you can mitigate the power requirements by getting say GTX560's, and its been a while since I worked it all out but for me its a reverse step, I want to use less power, get more points and run more stable for longer.

Its not all theory either, before this one I built weebeastie:

i7 970 @ 4.2Ghz (12 threads)
4 x GTX 470 @ 700/1400
Silverstone 1500W PSU
Asus P6T7WS SuperComputer motherboard.

PPD = 90-95K Max running bigadv on the cpu and 4 instances of gpu3.

Oh and in answer to the other question, this is the old design HF:

 

[Biffa]

Nice builds
would you have any advice for a person who wants to get into folding?

We have a forum just for that http://forums.overclockers.co.uk/forumdisplay.php?f=39

 

[Biffa]

Was measuring up the mobo tray for some extra standoff holes and maybe a few cable routing holes last night.

The missing mobo standoff holes are marked in yellow. Now with this much mobo hanging off them I want as many as I can get!

I ended up drilling the holes with a dremel, then screwing the brass motherboard standoffs right into the hole, and they just self tapped in.

Right radiator next, its going in the front, this rig doesn't need a heap of hard drives as it reads the VMware images off an iSCSI lun.

Lian Li radiator mounts, customised for 140mm size rad (drill a few holes)

Getting rid of all the junk in the 5.25 drive bay areay meant removing a lot of screws, and they are always in hard to get at places.

Slid right in there no problems.

 

[Biffa]

Yup, just been away

 

[Biffa]

There is a wee bit of a problem with that, I don't want to mod the case that much and there are bits of aluminium "guides" sticking out in the HDD rack just in the way of putting the radiator there.

What might be a better idea is to mount the fans on the radiator rather than leaving them in the door. For now its cool enough but come summer I might be thinking of "alternatives"

Sorry for the lack of updates.

OK so its been finished for a while, it doesn't look any different from stock externally but here is an internal shot of it all together:

 

[Biffa]

Last years prices

CPU's £633.2 x2 = £1266.4
Case = £259.97
Mobo = £417.00
Mobo Block = £72.33
CPU Blocks = £47.16 x 2 = £103.67
Res = £49.00
Fans = £20.43
Radiator = £49.00
Fittings, tubing etc = £61

£2,306.83

But I raised around a grand selling off a bunch of old kit, so out of pocket cost was around a £1,300.

At full chat it pulls around 630W from the wall. Compared to my last project its about half the power.

 

[Biffa]

yet another amazing build i have seen tonight, /drool, very nice!

and whats folding btw? XD

Cheers!

In answer to your query:


Folding@Home is a distributed network program you can find out more info on how to participate in this very forum here

 

[Biffa]

Well done!

Best to post any questions about it here so its on topic

 

[Biffa]

Depends completely on what you want to do, and whether the application scales more than 4 threads.

So rendering, measured with something like CineBench 10 shows +50% improvement in rendering times.

But games show little or even negative improvement, they show little improvement going from 2 to 4 or 4 to 6 cores so 12/24 isn't going to make any difference

But as a server or distributed computing or scientific computing machine, thats where multi cpus and especially the SR-2 because it can be overclocked to improve results even more

 

[Biffa]

Well seeing as the Socket B2 (LGA 1356) Sandy Bridge-EP Xeons aren't due till Q4-2011 we might be in for a long wait, and thats for 2 sockets.

It'd be interesting what the quad socket Socket R (LGA 2011) Sandy Bridge-EX (8/16) scales up like, but probably mad expensive unless you can get dirt cheap ones on the bay like BaBa