Project Nostromo

[Biffa]

Project Nostromo

Replacement rig for my SR-2 which has done a sterling job as my primary VMware work machine, running multiple VM's while folding away for the Worlds Greatest Team<tm>.

But I wanted to optimise the performance whilst maintaining or lowering my power usage in the office. Plus I wanted to test overclocking the G34 Opterons.

This is my hobby, my sickness, so if you feel the need to express your disgust at all things DC then please don't expect me to respond

This isn't my first WC Opteron rig, but its going to be a bit more special hopefully than the last and I am going to try and resolve some issues that have arisen from that build in this one.

So this really will be a work in progress, I haven't started yet, I've just been waiting for all the parts to arrive and a suitable evening to kick off the build.

But to begin with here is the plan and some pics of parts that have arrived.

Motherboard: SuperMicro H8QGi+-F
CPU's: 4 x Opteron 6166 HE 1.80 GHz - Socket G34 LGA-1944 - Dodeca-core (12 Core) - 12 MB Cache
Memory: 32GB DDR3 (16 x 2GB) 1333Mhz Low Voltage Ram
Case: Xigmatek Elysium Full Tower Case - Black

Cooling:
Radiators: Swiftech - 1 x 360 + 1 x 240
Pump: D5 + XSPC res or DDC 18w + XSPC Res (Have both to try)
Blocks: Aquacomputer Kryos G34
Fans: GT1850's
Fittings: Various compression and barbs, I'll decide as I go.

This rig will be optimized for cooling, noise is secondary, but reasonably important. We will see this time how it goes

Pics:
Parts arriving:

Block fitting plans...
45° fittings fit ok:

0° Fittings no good

Combo 0+45° fittings might work..

Hmm.. 45° compressions have smaller internal bore through the turn could be restrictive.

Maybe better to go for barbs.. no restriction and fit fine..

 

[Biffa]

Om nom nom nom nom nom.

Strange RAM choice though. 2gb sticks in this day and age?

You need to populate 4 slots in each bank to get the optimum memory performance this was just the cheapest way to do that. Plus i couldn't find low volt 1gb sticks anywhere

What performance boost do you get from Haribo, apart from a sticky mess in the watercooling pipes?

The performance of keeping the kids out of my hair

 

[Biffa]

Update: All the parts are now here, managed to snap a few shots before heading off this morning.

Blocks, cpu's, Pump/res and PSU, and something else ... New toy

CPU closeups:

Elyslium box is a monster!.......... But whats reallyscary is the size of the mobo box

Not so big a difference in size between the P80 and the Elysium. Well not from the outside at least!

THE MOBO UNPACKING CEREMONY!!!!
Oooh..... Ahhh....

Oh..

Ooooh.... er.. pink!

Now THATS more like it!

 

[Biffa]

Thanks guys, sorry for the lack of updates, but between building issues, and life/work I don't get as much time to post as I used to.

But I HAVE ZE UPDATEZ! Sit back and enjoy.

 

[Biffa]

I don't seem to see the updates?

LOL I'm doing this old skool, as I go along, this is no cut and paste from some "OTHER" web site, this is an OcUK Excloosive

Here she is in all her nekkid glory!

Having slammed in the RAM

Now some of you may ask, why, after spending all that would you get cheapo eco ram? Well firstly, this was all done on a budget, CPU's are ebay specials, mobo was a steal compared to what it normally costs, and 32GB of RAM isn't cheap in anyones money. That being said, it will be interesting to see how it goes with this RAM, it ticks a lot of boxes, its Corsair, its 1.5V and its the right speed, there isn't a lot of overclocking headroom on these chips but what there is may be limited by this RAM, we will have to see

Forget fancy TIM cleaner, get this stuff, I've been using it for years its just well awesome for everything. This one can has lasted me forever and its still half full! I know I bang on about it but it just is the best. Oh and excuse the covered up competitors names.

I give you the humble G34 socket! Home to a cpu, but more much more

CPU's in.

Next.. water block Unpacking!

 

[Biffa]

I got a confession to make, I love Aquacomputer stuff, its just different, looks good and just works for me. Dunno, its simple elegance. Yes the Kryos blocks are spoiled a little by the G34 mounts but theres not too much you can do about that unfortunately, I think most AMD block mounts look poo compared to Intel ones tbh.. anyway on with the show.

Block come out of the box and after you rip open the bubble rap like a small child at Christmas, this is what you find. A nice vacuum sealed and packed block.

Also in the box, along with the nuts and bolts and springs, they have thoughtfully provided some of the best TIM EVER, some of this!

Not the easiest to spread mind, but with these CPU's a rice sized grain in the middle is all you need.

NOMNOMNOMNOM... SHINY SHINY.. as I said, nice stuff Aquacomputer, nice

There, all tucked in nice and safe. Sorry no gratuitous screwing in shots, I'm sure its nothing you all haven't seen before. The bolts just go into the threaded holes in the backplate that comes with the mobo.

And there they all are, like a flock of Space Invaders

Here endeth the update.

 

[Biffa]

Excellent, glad to see this project taking off. Pity it wouldn't be much use as a gaming rig because I'd love to see what this can do compared to what I have now!

Fancy giving us a ballpark of how much a quad CPU, 32gb setup costs?

Yeah gaming isn't its forte, mind you with a game that likes multi-threading, and if I added a GTX680 in there for kicks and giggles.... who knows!

Cost.. hmm.. well I've sold my SR-2 mobo, ram, blocks and two X5650's so far from the old rig so that has offset quite a chunk of it. I think I got around £1200 for that lot.

The CPU's from "dove bay" were £240 each, and the motherboard was £450, memory was from our very own OcUk shop for £132.64

Had the PSU already, so then its just case, blocks, rads, fans tubing etc etc etc etc blah blah, nothing different there to any other build.

Again some of which I had, but I've changed to a different fitting size for this build, so they were all new. But I've got a lot of stuff to offload now from the old rig including case, rads, fans etc.

 

[Biffa]

Yeh might not be ideally suited, depends what kinda clock speed they'll be running at I guess?

I find it pretty shocking you got so little back for the SR2 kit! Must have cost over £3k only a year or so ago?

All I need to do was cover the cost of this baby

damn that looks immense Biffa, guessing it's for the F@H gig?

Subbed

Folding is its "other" job

 

[Biffa]

What seem odd to me is that the CPUs look rectangular rather than square and the waterblocks look rectangular rather than square, but the waterblock then fits on so that the longest edge is going across the shortest edge of the CPU, if you see what I mean.

Also, subbed.

Yes looks that way certainly, however although the heatspreader on the CPU is rectangular the block is square, slightly narrower than the heatspreader is long, and slightly wider than it as well. Actually the block body is the standard Kryo one, is just the fittings that have been changed. The blocks were actually the hardest thing to get hold of, took ages.

 

[Biffa]

I run virtual machines on it with fairly low loads.

 

[Biffa]

LOL sorry fell asleep last night before I got a chance.

OK Folks.. back in the saddle, this should be the final stretch


First some case pr0n.. when this case was announced, I liked its looks, some feel its a marmite case, but I like it.
Close-up of the top outlet cover closed.....And... open!

Oooh power button & industrial bolt heads......

Where I had "planned" on putting the 360 Rad (see later) but regardless a nice filtered top.

My new spare room.

Planning the bottom rad mount points:

I was going to put the fans under the rad, but as you can see in the next photo, it would have blocked ALL the outlets on the PSU, so I reversed and put the fans on the top pushing air out.

Test mounting the fans on the 360 rad became a challenge when I realised that some of the holes had not been threaded which led to me trying to force a screw into the hole to rethread, but ended up with me almost ruining the radiator.

Luckily I didn't hole the rad, and after some high pressure leak testing, I went out and bought a thread tapper and just got on with it

 

[Biffa]

Now it was time to fit the motherboard, I knew that I'd have to drill some extra mount holes as no case comes with the right number or position of holes for these motherboards.

But as you can see, my plan of putting even a 360 in the roof was doomed from the start so it would be back to the plan of putting it in the front drive bays.

So initial fitting, note how high up the mobo goes in the case. Thats why when you pick a case for these you have to have enough space for a second PSU above the IO plate.

I had to drill and tap about 5 or so holes, I forget exactly how many, the red dots are the ones I think I had to do, but there may have been one more, but who's counting! All I know is it is tougher doing it on steel cases than on Aluminium ones

The top of the mobo tray area is recessed, so I had to come up with something that worked there, luckily I had enough standoffs and nuts to make it work. Yes thats one nut and three standoffs.

In the pictures above you can also see the extra heatsinks I added to many of the mosfets on the board, thats because traditionally these motherboards are designed to be in a 1U chassis with a gazillion 40mm San Ace 10K cooling fans screaming their heads off forcing air across the motheraboard, through the heatsinks and out the back in an airconditioned datacenter. Well as you can imagine thats not the plan here, however there is decent airflow in the Elysium with the door fan and the design of the airflow I am going to use, so to add extra cooling where needed, some gpu memory heatsinks were used because a) they are cheap and b) effective


Next was fitting the 360 rad, fitting in the top was off, so the front it was, it went quite well, using EK brackets, I managed to fit it so that I could get one of the drive cages AND my fan controller all in together, all that was compromised was the fan in the drive cage, but I would use that later!

Rad fitted from the front and inside:

Inside looking at the fan controller fitted in front of the drive cage, and try in shot with everything back together to make sure it fits.

 

[Biffa]

Last update Putting it all together.

So here we are, everything in, fittings on, ready to start the tubing planning. Note the two fans up top which are exhausts? They are mounted as far away from the motherboard as physically possible, but I couldn't put one in the middle because it would get in the way of the tubing coming out of the top cpu block.

Close up of the top of the mobo area:

The 240 rad in the bottom (fans set to exhaust)

Final tubing plan:
Res -> pump -> CPU0 -> CPU3 -> CPU2 -> CPU1 -> 360 Rad -> 240 Rad -> Res
This gives a best compromise, as otherwise the top CPU only gets reached by the time the water has gone through three constrictive blocks plus gravity working against it to get there. This way its only the second block in the loop and then gravity is helping to get to the other blocks, the rads and back.

Some close ups:

Risky pump location, but hey, I'm livin on the edge!

 

[Biffa]

Heh so I lied, sorry wasn't the last update.

So leak testing shots and some videos!

First leak testing with pure distilled, also helps blow out any bits of crud left anywhere and helps identify any difficult air traps that I'm going to have to work on later.

Some videos, first a video of the airlocks being chased around the tubing, took ages to get them all out

Eventually was left with a few bigger bubbles working their way out as I tipped the case around.

So after the successful leak test (which did show up a slight leak around the pump, fixed with a few turns of the screws) then its in with the fluid I'm going to use.

This is Nanoxia Hyperzero ReadyFluid - Clear Protect, its clear which I like, and its anti corrosion and anti algae properties should be put to the test nicely in this situation

Its white from all the bubbles, and it made a load of froth, but eventually it all settled down.

I'll put some finished shots up soon, but currently its overclocked 25%, so pretty much 2.3Ghz out of 1.8Ghz chips and stable just bumping the volts from 0.9 to 1.0375V

 

[Biffa]

Excellent work dude. It's looking rather sick! Is that gong to be the 24/7 clock or will you push it further?

Problem is memory, at up to 250 base clock it runs at 1066 speed + the OC so 1066 x 250 / 200 = 1332.5Mhz and I don't think the DIMMs I use like going anywhere above 1333Mhz Faster DIMMS and some good chips and you shuould be ok up to 260-265

I'm happy that I bought £240 CPUs and am running them at £700+ CPU speeds at lower volts is good enough for me.

Excellent work there. I'm keeping my eye out for a H8QGi+-F board - I'd like to know how easy the process of flashing the oc bios was and then the actual process of overclocking.

Any screens or videos of that?

That's one hell of a machine you've managed to build there....

Its all tear at [H]ardforum's fault hes the one who created the OC'ed bios and all the instructions you need are in that thread

Supermicro H8QGi/6 and H8QGL Next Generation OC BIOS

 

[Biffa]

Spread is about 3°C but its not the 4th CPU (CPU3) that is the warmest, its the 2nd:

CPU0: 44
CPU1: 48
CPU2: 45
CPU3: 46

Because if you look at the loop CPU1 is the last one in the loop.

 

[Biffa]

It varies, usually a closed loop equalizes in temperature after a while, but these blocks are quite restrictive. Currently its 45-47°C so a variance of 1-2 degrees across the cores is pretty even. But I've seen much greater variance where I just went CPU0-CPU3 in that order, the temp difference was nearly 10°C esp when the blocks gunked up.

 

[Biffa]

I think the guys who are running 1600 memory are able to reach higher base clock speeds up to around 260 but I am only aware of one other guy with the same CPU chips as mine that has done that. But buying 16 sticks of high speed memory with tighter timings for that extra boost in speed is a big expense and I am well pleased with what I was able to achieve with the memory I had

So not its not a limitation of the board, but CPU wise they have limits to how high you can push the volts and those limits are hard coded into the chips, so with mine 1.050V is the highest they will go, but I'm running the overclock at one step lower 1.0375V so there is a little headroom left on the CPUs probably.

 

[Biffa]

Heh ok I'll try and get some shots, its just that its not in a very photogenic location