XXL_Folding

[Stan_Lite]

Been meaning to to an SR-2 build for ages and am now finally getting around to it. Bit of background first. I'm currently Folding on my gaming rig which has a 3770K and 3x GTX 670s. I recently bought a GTX 690 from Biffa in the MM to put in there and needed somewhere to put the 670s. Around the same time, Kerrgreg was offering SR-2s in the MM at very reasonable prices, so I decided to bite the bullet and bought one with MB waterblock and RAM. I've since added to that 2x Xeon 5620s with waterblocks. I've also managed to pick up a Xigmatek Elysium case from the MM at a good price as well as a 128GB Crucial M4 SSD.

List of items so far:

1. SR-2 MB
2. SR-2 MB waterblock (EK)
3. 3X GTX 670
4. 12GB Corsair 1600MHz C9 RAM
5. 2x Xeon 5620 CPUs
6. 2X Koolance 370SI waterblocks
7. Crucial M4 128GB SSD
8. Xigmatek Elysium case (due to arrive on Tuesday)

Still to get:

1. EVGA SuperNOVA NEX1500 Classified PSU
2. 2X Xeon 5690
3. Watercooling stuff as required
4. Upgraded GPUs

The plan is to build the rig with the 5620s and 670s at the moment and then to upgrade to 5690s and either; 4X 690s or 4X Titans (whichever yields the highest PPD for F@H). For now I will be watercooling the CPUs and MB with a normal WC set up and the 670s will be stock cooled. Once I get the 5690s and upgraded graphics cards, I will be building some sort of fancy cooling system using the Hailea water chiller I bought for a previous project. This will be some time in the future as I have 2 holidays booked and need to get my credit card cleared so it will be later in the year before I get any of the upgrades done.

I'm waiting until the case arrives on Tuesday before deciding how the WC loop(s) will work and I'll be ordering the PSU at the same time as I get the WC stuff. I was keen to do this on a budget since I got most of the bits so far from the MM but I never realised until I got the MB what a power hungry SOB the SR-2 is. I intended to use the OCZ ZX1250 from my gaming rig in this build but, while it has enough grunt, it has nowhere near enough power connectors. The SR-2 needs 2X 8 pin MB connectors plus 2X 6 pin PCI-E connectors (1 for each CPU) as well as another 6 pin for the MB if you're running multiple GPUs
There are cheaper options than the EVGA PSU but it is built for the job and also has fully braided modular cables - which will save me money for cable braiding stuff and a hell of a lot of time.

Anyway, enough of that and on with the pictures.

The motherboard. The photo doesn't do justice to the size of this thing - it's huge.


That's 2 CPUs, count them, 2.


Again, count them - that's 7 PCI-E slots (4 of them at X16).


Token arty shot:


Motherboard blocks:


CPU blocks and backplates:


RAM:

 

[Stan_Lite]

First things first. Remove the old backplates...


... and replace with the new:


Next, fit the main MB block:


And then the Mosfet block:


CPU blocks fitted:


RAM fitted:


So far so good. I've done as much as I can for now. When the case arrives on Tuesday, I'll test fit the board and see where I'm going to put radiators, pumps, reservoirs etc and then I can order anything else I need.

The board so far:

 

[Stan_Lite]

Been a delay with the case so I'll post a few pics of the PSU which arrived yesterday. It's a 1500W PSU but can actually be 'overclocked' to supply 1650W. It comes with its own software as well (there's a USB cable to connect it to the mobo).

Side on - it has a handle


End on - no shortage of connections:


Hefty power cable. Pity about the proprietary fitting:


Motherboard cables:


Sata, molex and miscellaneous cables:


Bright red PCI-E cables - 16 of them


All cables factory braided - very neat


And finally, also arriving yesterday, Windows Ultimate. Dual CPUs requires Windows Pro or Ultimate and Ultimate was only £20 more so I got that. Not that keen on Windows 8 so far so I'll stick with Windows 7 on this one and see what the upcoming update does to 8.


That's it for now. I've ordered some watercooling stuff so I'll post up some pics of that as and when it arrives and hopefully the case will get here soon and I can get on with it.

@ Kerrgreg: I'll be giving all the blocks a good flushing before I set up the loop properly

 

[Stan_Lite]

That's no proprietary fitting on the PSU, it's a C19, needed for the (no doubt) massive 100% load amperage that PSU could levvy.

I stand corrected. That's good to know in case I need a replacement

 

[Stan_Lite]

Epic-ness! That was the best PSU you could have got for this machine!

It appears to have been designed specifically for the SR-2 with the exact amount of motherboard connectors in exactly the right configuration as well as more than enough PCI-E cables to populate all 7 slots if need be. All of the cables are extra long as well to suit the size of the board and the sort of gargantuan case required to house the thing. Damned expensive but worth it really.

 

[Stan_Lite]

looks absolutely epic, how much would this cost ?

I've just been totting it up roughly and to build it from new parts to the current spec, you'd be looking at about £3,200. I've saved quite a bit by buying secondhand stuff - probably about a grand. I also had the graphics cards already so the build is costing me a about £1,200 - £1,300. When it gets the upgraded Xeons and graphics cards in the future, you could add another £5,000
Thinking about it, as much as it would be nice to have such a huge e-penis, it doesn't make sense to get the high end upgrades. I could probably build a couple of 4P server units for the same money, which would give me much more Folding PPD. Might just get a 4th 670 and watercooling blocks and spend my money more wisely.

In other news, the case arrived yesterday but I haven't got much done with it yet. I'll get some stuff done with it today and post pics. Unfortunately, I forgot about the Easter weekend when I was ordering my watercooling stuff and, while it was dispatched yesterday, it won't arrive until Tuesday (as will the items I ordered from OcUK, although they're mostly for the changes to the gaming rig - except the extra 12GB RAM I ordered for this build ).

 

[Stan_Lite]

Got a few bits done yesterday now that the case is here. The first shot was me having put the board in and trying put a few cables.


At this stage, I noticed a couple of things that annoyed me. Firstly with the case, in the top right hand corner, you'll see a mess of multicoloured cables from the front io panel. There is no other way to route these than through that top grommet. Looks really messy I also don't like the USB3 cables that have to be routed out the back into the rear io panel but that isn't confined to this case.


Secondly, on the board, the HD audio socket is in a really stupid place, tucked in the middle of the PCI-E slots. I should be able to make it reasonably neat - the graphics cards should just about squeeze by it but it should be elsewhere.


I had a brainwave about the messy io cables. I was toying with getting some of that spiral cable wrap stuff but that's not the neatest looking stuff either and then I happened to notice a bit of spare black 1/2" hose I used in a previous build. I cut a couple of lengths to size and made slits up the back and shoved the wires inside. Looks a lot neater now.


Where I've got to so far:


This is as much as I can do for now until the rest of the parts arrive on Tuesday. I've fitted as many bits and cables as I can and fitted an extra 120.1 radiator I had lying around. The plan for the cooling loop is to have a bay res/pump combo in a couple of the lower 5.25 bays (I've ordered a res to fit one of the D5 pumps I have in my collection). It'll go from there to an Alphacool NexXxoS UT60 radiator in the top with the barbs at the back of the case. It'll go from there to first CPU (on the left); from there to the second CPU; then down to the 120 rad before heading up to the main MB block and then on to the mosfet block and finally returning to the res. You'll have to imagine all that in your head as I'm no good at drawing diagrams

A few final images.

Plenty of room up top. Not sure if I could squeeze in push-pull with that thick rad. If not, I can just go with 3 fans or mount some on top of the case.


Towards the front where the res and fan controller still have to go. That DVD drive and SSD won't necessarily be staying exactly there. I'll wait and see how things fit in before finding a final place for them.


And a final shot towards the back of the case. You can't really see from the pic but I've neatened up the HD audio cable and managed to tuck it into a place where I should be able to fit the graphics cards around it.


Some of the eagle-eyed among you may have noticed that there is a 1/2" hose from the second CPU block to the bottom rad. This is just in place to see how it fits and will be replaced with 7/16" hose when it arrives. Also, fans. There will be 2x120mm fans blowing in from the HDD bays, the bottom rad is blowing into the case and there will be a 200mm fan blowing in from the side over the graphics cards. The 360 rad will be venting from the case as will the 140mm fan at the back.

Making progress but frustrating having to wait.

I'm off up to Falkirk for the Larbert beer festival in a couple of hours to chill out. When I get back tomorrow, I'll tidy up some of the carnage that is the inevitable result of me building PCs.
The dining room table:


Screws and stuff on the chest of drawers in the dining room:


And the spare settee in the living room:

 

[Stan_Lite]

I forgot to say thank you for all the kind comments (although most of them have been in praise of the awesome hardware and not my incredible system building skills ).

I forgot to reply to TheDesX77 about the Tesla cards. Apparently they're no better for Folding than standard GPUs and much more expensive. I assume the Folding clients are optimised for standard GPUs as that's what the majority of users will have.

 

[Stan_Lite]

Good work stan!

Personally I opted for running the PCI-E power and cpu Power from the top of the board then its not in the way of the gpus when you plug them in

The cable isn't long enough to reach from the top so I'd need extensions which would probably be a bit messy. I should be able to work around it the way it is.

Your doing it all wrong ...if you can work on a computer the day after a beer festival ... your doing it all wrong

Great build btw, cant wait to see it finished.

I wasn't actually going to be working on the build, I was only going to tidy up. Even that proved too much and still hasn't been done

Guys, I was only kidding about the system builder thing, lol. I'm actually a bit rough normally. Due to delays getting hardware, I'm being forced to take my time on this one so it's going a bit neater than usual and with more planning. Usually I get all the bits in one go, bung them together in an hour and switch on. The results are usually pretty rough.

I'm thinking this is the better way to do it, bit by bit with plenty of time and patience. Having said that, assuming everything comes tomorrow, I'll only have 2 1/2 days to finish it as I'm off on holiday on Friday and only have half a day when I get back before I'm off to work for a month. I want to get this up and Folding by Thursday.

 

[Stan_Lite]

Folding with an SR-2. My good sir, I salute you!

If you need to get rid one of the GTX670s, I still need one for my folding build

Good luck with this build. It looks very promissing.

One thing though; I see a lot of non sleeved cables. Even folding builds have to look good

Thank you sir. I haven't fully decided what I'm going to be doing with the 670s yet. If I do get rid of them, it won't be for a while yet anyway.

This is nowhere near as nice a build as your beautiful looking Folding rig. I've been keeping an eye on your thread and I'm hugely impressed by the attention to detail.
Most of the unsleeved cables will be hidden. About the only ones that will be visible will be the SATA cables into the mobo. I might look into braiding them in the future but not at the moment. Could be a bit tricky as a couple of them are hard-wired into the front io panel.

 

[Stan_Lite]

Only a small update today. My watercooling bits didn't arrive. Apparently they were dispatched on Thursday at 10:00 and, until a couple of hours ago, Parcelforce were aware the parcel was coming but hadn't received it I contacted the supplier to ask what was happening but haven't received a reply yet. The Parcelforce status has since updated to ".... arrived at Manchester Mail Centre on the 2013-04-02 and we are currently processing it."

No sign of it having moved from Manchester yet. Hopefully it will move overnight and get delivered tomorrow. If not and it doesn't arrive until Thursday, I'm going to be hard pushed to get the thing finished before I head off on holiday on Friday (I leave at about 06:00 on Friday so need to get done not too late on Thursday). If the stuff arrives tomorrow, I'll get it all built tomorrow and test the loop overnight and spend Thursday overclocking and testing/benching before getting it Folding.

Anyway. At least the stuff from OcUK arrived - at least someone is dependable (it's just a pity their watercooling stock isn't as good as I would like). I received the extra RAM I ordered and have bunged it in - looks a lot less bare now


Now 24GB in there. Overkill I know but, that's what this baby is all about really.

Also received a jumbo tube of MX-4 (pictured next to the ubiquitous Haribo for scale):


That should keep me going for a bit
I took out the CPU blocks and re-did them with the new MX-4 as I had originally done them with some old AS-5 I had lying around. It was a few years old so I wanted to make sure there was good gunk in there. I didn't redo the MB blocks as they were done with the last of the MX-3 I had which was a lot newer (they're also very fiddly to get on and off).

Once the watercooling bits come, I shouldn't be too long getting her finished. the waiting is really frustrating - especially when I'm getting pushed for time

 

[Stan_Lite]

What are you filling the loop with?

Alphacool ultra pure water for now. I won't be using any dyes as I'll probably be changing the loop if/when I watercool the GPUs.

 

[Stan_Lite]

Well, it looks like this build will be at a standstill for a while. The watercooling parts are still being "processed" in Manchester (that's the better part of 2 days now) and I'm off on holiday tomorrow. I can only assume "processing" translates as "sitting in a corner until there's enough to fill a van". I hate Parcelforce

I get back from holiday on Monday the 15th but I'm due to head back out to Egypt to work the next day so I won't have time to do anything meaningful in that short a time frame so I probably won't get anything done with it until I get back on the 17th of May

 

[Stan_Lite]

gutted to hear this... what parts are you waiting for stan?

Lots of stuff. Radiator, reservoir, fan controller, water, kill coil, hose and loads of fittings If it had just been a few bits and pieces, I might have been able to get something locally to get it up and running for now but it'd cost a fortune - even if I could get all of it locally (which I doubt).

 

[Stan_Lite]

Sad times, i probs have most of that spare, admittedly all second hand

I could probably cobble something together with bits I have but it would look awful and would need to be stripped down and redone anyway so I didn't see much point.

This looks absolutely fantastic! I'd love to do something like this, but i've gone from a larger build to a smaller build anyway and this would blow my old build out of the water!!

I fully intended doing a smaller build this time (Prodigy gaming rig or the like) but the chance at doing an SR-2 build relatively cheaply was impossible to resist

Although I can't get anything done, I have managed to sort this...



...into this.


Two of the most tedious hours I've ever spent but worth it as I'll now be able to find what I need a lot quicker than before. I've also discovered that my family will be self sufficient in PC component screws for several generations

 

[Stan_Lite]

Not really an update as such but I got back from holiday today and the watercooling stuff has arrived while I was away (nice having a neighbour who'll take stuff in - thanks Mrs H).



Pretty much all I need to finish the build but, since I can't do anything now until I get back from work in a month, I'll order some new fans while I'm away. I've measured it up and I can just squeeze the radiator in with fans either side. Is there any real benefit in a push pull setup? Are the temps much different? I believe the 1450rpm Gentle Typhoons are the flavour of the month for radiator fans at the moment.

 

[Stan_Lite]

Push-pull does help, but not by a huge amount. It is all in %s and degrees. If you have room for it do it, if you don't it will be fine.

You doing the Chimp Challenge atm?

As I said, there should be just about enough room. I may as well do it. My limited research suggested what you say, that it makes a few degrees difference, but not essential.

Not doing the chimp challenge. Didin't even look at it to be honest

Really nice build there, can't wait to see it finished.
I did skim a bit, what colour coolant are you going for?

Can't wait to see it finished.

Will just be running clear to start off with. Once I get organised with watercooling the GPUs, I may well add some colour and some lights but that will be a different setup altogether so I'll cross that bridge when I come to it.

Love an Sr-2 build, I've come close to buying one many many times.

What's your final rad layout going to be?

For the moment it will be the 360 in the top with the barbs towards the back of the case and a 120 in the bottom. Loop will be res/pump -> 360 rad -> CPU1 -> CPU2 -> 120 rad -> main MB block -> mosfet block -> res.

As I said in the OP, the long term plan is to WC the GPUs (4 of them) and cool the whole thing with a Hailea water chiller via a cunning and unusual setup I'm devising - details of which will be forthcoming when and if it comes to fruition

 

[Stan_Lite]

Since I'm away at work, I have had plenty of time to think about things and I had pretty much decided to go straight into phase 2 of the build when I get back in 3 weeks time. I was looking at what I would need to get the two loops hooked up to the water chiller and planning what to buy from where. That was until I heard the very strong rumours that the Nvidia 700 series were due to be released in May. It's back to phase one now until I see what the new 700 cards bring.

I had made up my mind that I would go the middle route and get 4x680s for the build but I'll wait and see now. If the 780s are as good as they are rumoured to be and reasonably priced, I'll get those (once there are waterblocks available for them). Alternatively, the price of 680s may well drop and I might get them as planned. There's also a good chance that the MM will be flooded with 680s when the new cards come out so that's a third option.

So, it looks like I'll be going ahead as planned when I get home. I'll get the watercooling loop for the CPUs/MB sorted and bung in the 3 670s for now and bung the 690 into the gaming rig. As a little offshoot of this build, I'll also be doing up the gaming rig. I currently have the CPU watercooled in that with the 3x670s running with stock coolers. I'll be putting a second loop in there for the 690 and maybe one of the 670s for a Physx card. I'll be doing that after I've completed the main build and will probably tack it on to the end of this log as it doesn't really merit a project log of its own.

 

[Stan_Lite]

stan, seeing as loudb is adding another 4P e5 set up to reduce electric costs, see if he wants to off load any of his 680's cheap

I'm going to wait and see what the 700 series are like before I make any decision. I did read a rumour that the 780 would be within 10-15% of the Titan. At the moment, it's all rumour so I prefer to wait until they come out and have been reviewed and used for real.

WHat will you use the gfx cards for? Just for folding? or will you also occasionally game with it?

It will be pretty much all folding. I'll have the gaming rig with the 690 for gaming, which should see me good for a few years yet. To be honest, I don't do a hell of a lot of gaming but what little I do, I like to do at decent quality.

 

[Stan_Lite]

Isn't it more economical to build a 4P Opteron system than to add 4 GTX680s? In terms of power usage and maybe even building cost.

Absolutely. No question about it.

Don't get me wrong. 4 680s stacked and watercooled is very sexy looking and will also fold quite nicely. Just making conversation

That's the only reason for doing it.

As I said in a post above, I never actually intended building a system like this. My next project was going to be a small gaming rig and then I was going to build a 4p Folding rig or two. Unfortunately, I got sidetracked by a cheap board etc and here I am. The way I look at it is, if I'm going to do this, I'm going to make a job of it. I'll still build the 4p rig(s), just not for a while yet.

I don't mind you asking the questions - hell, I'd probably ask the same if it was somebody else. This is one of those projects I've wanted to do since the SR-2 board first appeared. Once I've got it out of my system, I can get back to the relative sanity of building sensible () quad socket Opteron/Xeon systems