Home Lab Threadripper Build Thread.

Vince

Vince

Vince

Man of Honour
Joined
30 Oct 2003
Posts
13,634
Location
Essex
I was watching the release of threadripper carefully and the core count got me thinking about how I could use Threadripper to replicate parts of my works virtual infrastructure in order to plan out projects more effectively and simulate infrastructure and software upgrades outside of the building, which frankly isn’t really possible on the ageing 1090T system I have at home now. To give some perspective I guess we should start with what I do for work in my day job which is basically to head up all technical projects for an Intellectual Property Legal500 firm. I have been at this job now for 5 years which is a long time and in that time we have pretty much had a top to bottom rebuild and our infrastructure is better than ever, I’ll talk a little bit about what I manage before we get to the build.

Without going into too much detail split between 2 sites we have 6 DL380 G8’s, 5 of which are esxi hosts each rocking 192gb of memory and 2x6 Core HT Xeons, a huge (for the size of the company) HP6300 EVA with 5 trays of 600gb SAS disks and two HP store once 2400’s. This is all connected up with fibre and powered by a 30k HP UPS installation that weighs in at 2 metric tonnes. I won’t touch too much on switching or security but our core switches are layer 3 POE cisco units and we run mainly FortiGates on the perimeter.

It’s pretty clear that x399 isn’t going to get anywhere near the performance metrics that I am getting out of the infrastructure above but the core count coupled with the I/O should be able to offer me the ability to effectively replicate subsets of it while also giving some insight into projected epyc performance in for example SQL,IDOL,SharePoint,iManage and other similar corporate workloads.

This build from top to bottom will disappoint some but hopefully interest others and I might even be able to answer some questions along the way. I’m not totally against giving some gaming benchmarks and a little overclocking but that certainly will not be the focus of this thread so let’s not steer it that way too much please gents. To go alongside what I am doing with this kit I have convinced a buddy of mine (didn’t take much convincing tbh) to come over and experiment with some of his creative workflows which he tells me involves the following: Lightroom 6, LRTimelapsev5.7, aftereffects 2015 cc, premiere pro 2015 cc, Gopro Cineform decoders, Avid codecs etc. I will also share his thoughts or perhaps even get him to chime in.

It’s about time I stopped waffling on so let’s get started, the kit list I have put together is as follows, it’s not all here yet but I have received the first bit today which is the PSU and ordered the rest so thought I would get started putting something together to document it on here.


The Machine:

Gigabyte AMD X399 AORUS GAMING 7 Gaming Motherboard Edit: AsRock Taichi x399 Edit: Asus Prime

AMD RYZEN Threadripper 1950X 16 Core TR4 Processor

2X G.Skill Trident Z RGB 32GB Kit DDR4 3466MHz RAM

Corsair Hydro Series H100i v2 Extreme Performance Liquid CPU Cooler

Crystal Series 460X RGB Compact ATX Mid-Tower Case (hope this is going to be big enough)

Seasonic 860W Platinum Power Supply

A solitary XFX RX480 8GB

256GB Samsung Pro NVME

4x1tb, 1x8tb WD RED, 3 or 4 random SSD’s probably in raid 0

Netgear ReadyNas Serving as extra LUN storage (perhaps snapshots, haven’t decided yet)

A USB stick for ESXI.

Fortinet Fortigate 30E (because I am poor and can’t afford the top end kit)

Cyberpower Rack Mount LCD Series PR3000 UPS.


I guess that’s really all I have for now. I will update as I start building and as I receive more kit which looks to be in the next few days.
 
Last edited:
In my experience, you'd be better off buying a matched 64GB kit than using 2x 32GB kits. Good spec though :)
 
EVH said:
In my experience, you'd be better off buying a matched 64GB kit than using 2x 32GB kits. Good spec though :)
Click to expand...

I haven't yet been burned by this yet and my last two sets have been non matched. Out of interest whats the difference as years back I bought a matched XMS 3 which just came in 2 packs in much the same way
 
Vince said:
I haven't yet been burned by this yet and my last two sets have been non matched. Out of interest whats the difference as years back I bought a matched XMS 3 which just came in 2 packs in much the same way
Click to expand...

The difference is that you're mixing kits, versus not mixing them. Mixing memory will have varied results depending on the platform, speed and density of the memory kits. Memory is binned at the timings and speed by the vendors at the density it is sold in. In combining kits you are eating into the guardband put in place by the memory vendor. Moreover, you're putting more strain on the IMC. 64GB @ 3466 on Thread Ripper is not likely to be plug and play, especially with combined kits. I'd fully expect to have to tune things manually, and even then I would probably expect to make some concessions.
 
Silent_Scone said:
The difference is that you're mixing kits, versus not mixing them. Mixing memory will have varied results depending on the platform, speed and density of the memory kits. Memory is binned at the timings and speed by the vendors at the density it is sold in. In combining kits you are eating into the guardband put in place by the memory vendor. Moreover, you're putting more strain on the IMC. 64GB @ 3466 on Thread Ripper is not likely to be plug and play, especially with combined kits. I'd fully expect to have to tune things manually, and even then I would probably expect to make some concessions.
Click to expand...

Fair play, seems reasonable. I have to be honest I don't expect plug and play and to be fair I originally I ordered a single 32gb kit but when I checked the order online they had added two, rather than argue I just went with it, because well, more is better right? I also figured that rather than just buy 3200, if I go a bit faster I might have a better chance of hitting 3200 at decent latency, not necessarily to hit advertised speed. I can asses the situation and then buy more if need be I guess.
 
Silent_Scone said:
Sounds good, as long as there's realistic expectations going in that's all that matters!
Click to expand...

Absolutely and that is exactly what I tell the wife :D.

My order is still not shipped :mad:. The board is assigned to my order but it appears its not left yet, hopefully it gets shipped tomorrow and we can get this show on the road. I'll throw a couple of photos of the PSU, Firewall & UPS up later tonight or tomorrow. With the firewall, other than routing, IDS, Antivirus and all the other good stuff a Fortigate can do I think I might have a play setting up an IPSEC tunnel back to the office passing h.323 so that I can set up an avaya ip phone or avaya sip DECT at home. That has to be a better solution than forwarding calls in to my mobile from the office.
 
I have also subbed. Be an interesting build, do not see many of these floating around with a different purpose.

Currently running a old x5650 with 48gig of Ram on my home lab. Struggle to justify the upgrade, but hoping you will talk me into it.

Do Forinet not do a VM appliance you could have used to keep the cost down?

Thanks
 
Likewise... Understand Ryzen now works (after patch) but on Linux KVM Ryzen has Nested Page Table problems and passthrough is not reliable/consistent so very interested to hear how TR works under ESXi.

Also interested in how that mobo splits devices into IOMMU groups both with and without an ACS Patch... so if you can boot direct into a Linux kernel and do some dumps (rather than into ESXi and then Linux) I'd be grateful...
 
tris179 said:
Do Forinet not do a VM appliance you could have used to keep the cost down?
Click to expand...

The do and it is something I was going to look closer into. The 30E for me was about baseline for what I needed and because I am going to switch between windows and ESXi I thought that the always up nature of a hardware firewall would probably suite my needs better than the virtual appliance. I paid about £330 for the 30E and that's about as far as I could stretch. The plus side as I see it is that the 30E runs pretty much an identical OS to the more expensive ent grade kit.

Methanoid said:
Likewise... Understand Ryzen now works (after patch) but on Linux KVM Ryzen has Nested Page Table problems and passthrough is not reliable/consistent so very interested to hear how TR works under ESXi.

Also interested in how that mobo splits devices into IOMMU groups both with and without an ACS Patch... so if you can boot direct into a Linux kernel and do some dumps (rather than into ESXi and then Linux) I'd be grateful...
Click to expand...

I am not going to lie some of this is straight over my head but I am more than happy to spend a little time learning myself and trying to provide you with some insight into this. As I start getting into the software side of the build I will make a list of the stuff people would like to know and hopefully with your help can answer some of the more complex questions :)

tris179 said:
I'll get this out the way now....

You do not need more than 8 GB's of RAM.... :p
Click to expand...

Only 8gb! - That might just be enough for one under provisioned SQL server :) Seriously I was actually thinking that the 64gb might be just a little short. Have you seen the price of memory these days? Its obscene.
 
Makes sense tbh, the firewall is the bit that you need up 100%..

Be interesting to see how this shapes up. You plan on chucking all the HD's in the same box?

If money was no object then 128GB would be a sure thing for me. :eek:

I personally like the all in one box approach, small foot print, multi purpose etc, but did you consider scaling out rather than up approach?

How do you currently replicate data (assuming that's what you do?) between to your two sites and do you plan on emulating that in your lab?

Just something else to bare in mind and maybe of interest to those subbed to this thread if you need some extra horse power for certain scenarios, check out http://www.ravellosystems.com
 
tris179 said:
You plan on chucking all the HD's in the same box?
snip...
I personally like the all in one box approach, small foot print, multi purpose etc, but did you consider scaling out rather than up approach?
Click to expand...

The plan is one box and storage on lan to back it up. The fact that the case has little room for hard disks was a school boy on my part but also largely down to my limited budget which went on the bits that really count, some getting creative and perhaps 3D printing of some case internals may be required. If need be ill buy a new case later down the line when I have more cash available. In terms of scaling the companies infrastructure, because of the nature of some of the data we keep we cannot use cloud services, we do however have a co-lo of our own kit in a secure data center. Put simply we need to design secure, robust infrastructure that scales well with the companies requirements.

tris179 said:
How do you currently replicate data (assuming that's what you do?) between to your two sites and do you plan on emulating that in your lab?
Click to expand...

We currently use dedicated HP StoreOnce 2400 appliances, they are basically dl380's running custom software with dedicated de-dupe, compression and cross site replication, used alongside Veeam they are very neat devices. I don't think there are any issues with sharing links as OCUK don't operate in this space so take a look here: https://www.hpe.com/uk/en/storage/storeonce.html any issues dons then please feel free to remove. This isn't really something I plan on replicating in the lab to be honest, firstly because the hardware is on the expensive side and secondly because I don't really need to replicate the whole environment to achieve the majority of my short term goals with this project. Some of the first things on my list bar getting esxi up and running would probably be an in place upgrade of legacy systems that are in desperate need of a revamp, we only have one of these which is our QlikView server and dashboard which is a dedicated box and not virtualised for some unknown reason, so that will have to be done straight away. Then I plan on simulating an iManage 9.3 to 10 + web upgrade and see what parts of our custom code fall to pieces.
 
I bought a new monitor for the build last night, nothing special as I'm running low on available funds but it is a 4k 28" freesync jobbie which I thought represented decent ish value for money. Should be better than my current 3x 1080p Samsung screens and I might just add another 2 over time. Also got my eyes open for another rx480 but what with the mining stuff going on they do not represent value for money right now so I may just hold on until a second card pops up at reasonable prices. The other option is vega 56 or something from the green team but for now I think that I am happy enough with what I have even if rx480 doesn't really stand a chance at these resolutions.
 
Given that updates are slow because of the motherboard stock situation I thought I would put a couple of teasers up of what is here so far :)









 
Last edited:
You must log in or register to reply here.
Back
Top Bottom