Home Lab Threadripper Build Thread.
- Thread starter Vince
- Start date
-
- Tags
- threadripper
Makes sense tbh, the firewall is the bit that you need up 100%..
Be interesting to see how this shapes up. You plan on chucking all the HD's in the same box?
If money was no object then 128GB would be a sure thing for me.
I personally like the all in one box approach, small foot print, multi purpose etc, but did you consider scaling out rather than up approach?
How do you currently replicate data (assuming that's what you do?) between to your two sites and do you plan on emulating that in your lab?
Just something else to bare in mind and maybe of interest to those subbed to this thread if you need some extra horse power for certain scenarios, check out http://www.ravellosystems.com
If cost is an issue I picked up from the a popular second had part selling site
16 cores 64gb xeon workstation
Dell t5600 £347
64gb ecc low power (4x 16g) ram £55
2x e5-2670s v2 (8 core sandy bridge xeons) £130 pair
Dell raid controller (forgot the model lsi esxi compat) £45
Swapped out the cd drive for a 4 bay sata 2.5" caddy £45
Purchased 2x sas to sata cables (will have 6 drives total) £12
Added 4x 240gb sata, 1x 120gb sata (boot) and 1x2 tb HD (all secondhand) £210
Got about £60 back from ram / cpus / hdds from the dell, and my old rig (dell t5500 dual hex 48gb) sold for £320
Pc3l keeps selling 2 cheap I'll upgrade when I see another set of 4x16gb going for sub £60
I took a gamble the low power ram and xeons 2670s were not on the compatibility list and I was advised it would probably not work but it all did!
I get a power config not supported warning as it want me to use a 825w psu (not the installed 650w) but I've had no issues in 3 months of 24x7 running.. The psus pop up for £60 but as I don't have an issue (other than the message on power up) I don't care
Soldato
That Gigabyte board does look good, though rather than getting 3 motherboards deep in TR, maybe wait and go Epyc instead!
I setup ESXi successfully on my Zenith as well. VGA passthrough is working fine, however, I can't get any USB devices recognized in the VM. So atm can only control through the ESXi console. Do I have to get PCI-e expansion card for USB then pass through that?
Shouldn't have to. USB on my board seems to just work. Once I have the VM up and running (with the usb controller added) in esxi I just mount the USB device by firing up the VM and then right clicking on the USB device right at the bottom of the console window and pressing connect. USB devices are then available through the VM. Can do a little video on this or a couple of screen shots if it helps?
Funny you should mention this. Temps etc are not available in esxi because of lack of support from the board. Ill have to get some screen shots up of this. I only know that it was fine as I have loaded it up for hours on end in windows and stressed it there. For example, I know that the temps with all cores under prime in windows reaches no more than about 70 degrees. I can only assume that we were hitting the same kind of temps in the VM stress tests. I will update later this evening though with some details.
I am actually fairly happy with throwing the money at the boards, worst case scenario is that I sell them on after I have had my wicked way with them. EPYC is tempting but at the end of it all it's not going to offer me any more (at least in the lab) than TR, TR perfectly suits my requirements in terms of core count and grunt so really for me, apart from the compatibility side of things, TR is perfectly adequate.
It's the features that I like, 8 sata ports (warning signs here), dual lan, 3x m.2, more pci-e than the 399-A. Just seems well rounded and much like the Taichi feels like pretty much the perfect candidate for a lab. The 399-A falls well short as its lacking in pretty much all areas compared to not only the Taichi but also all other x399 boards. Watch this space
If we have to go through every x399 board out there, we will, I will find the perfect lab board.
If you could post a few screen shots that would be helpful, thanks. I didn't see any options for this at the bottom of my console window but perhaps I'm overlooking something. I did add the controllers in my VM settings though.
I will post up some stuff later. Can't say for sure if it will be tonight as I need to tear the machine down again to get my gfx AIO properly seated. Spent hours last night trying to get the Vega card under an AIO and gave up really late with frankly not great results. Try again tonight I think.
Soldato
Fair enough, but it's vmware compatibility that made me suggest it, I have a TR system as my VMHost - but luckily I'm not limited to VMWare.
Just out of interest - what hardware are you planning on adding? I'd be very curious to know what combination of bits would exceed even the most basic X399 mobo's! I have a proper dual port Intel NIC, a SAS HBA, and GTX 1070, still leaves a x16, x8 and a x1 free at minimum (not at home to check!).
m.2 nvme drives, lots of them... the Prime has 2 slots, doesn't at this point support nvme raid and to be honest I just don't really like the board or Asus, yes its functional and works but it really is bottom of the barrel in terms of x399. When the Asus Fury Quad hits ill no doubt be adding more nvme than I really need. Put simply though I want a board more in line with the Taichi, more PCI-e, more sata, more M.2, More Nics, better layout. I didn't buy TR to be board limited yet that right now is how I feel.
Last edited:
Your right. You can do it that way in workstation and if I remember rightly you can also do it this way in the vsphere 5.5 client. Console in this 6.5 doesn't have the option. Why they can't just let us use a decent vsphere client is beyond me.
I'm done for tonight after fighting with and finally conceding to a gpu aio but will update as I did pass through a usb key to a vm when I was building that Windows 10 machine a few days ago I just don't remember exactly how. I will try and fire it all up again tomorrow night and update this thread with some more results.
Hello Vince, joined for you. I replied to the VMWare support but I think it's much better here.
Thank you for your really hard work, it's a time saver ! We are looking to test a build with ESXI too with Threadripper. Nothing is purchased yet, but we were looking for the Taichi board, however it seems it's a bit complicated and might be an issue for future ESXI updates.
With the Prime board, what are the disadvantages ? Only the missing features ? I've read about issues with error code AD, and post being way too long. Is the ASrock faster to boot ? What about the Gigabytes ? We are interested in the practical experience, ease of use, no lag at boot, easy BIOS, good RAM compatibility.
Thank you for your really hard work, it's a time saver ! We are looking to test a build with ESXI too with Threadripper. Nothing is purchased yet, but we were looking for the Taichi board, however it seems it's a bit complicated and might be an issue for future ESXI updates.
With the Prime board, what are the disadvantages ? Only the missing features ? I've read about issues with error code AD, and post being way too long. Is the ASrock faster to boot ? What about the Gigabytes ? We are interested in the practical experience, ease of use, no lag at boot, easy BIOS, good RAM compatibility.
Hello Gents, Been a while as I have been pretty busy with all kinds, mot's on cars, issues with BT that seem to have been dragging out forever, you know those issues where you make loads of progress then seem to take several steps back? That's basically the sum of my last two weeks. I have made quite a bit of progress in setting up my lab though, the fortigate is all installed and I have configured policy based VPN's with multiple phases between home and the London office, something that I needed to get done for a small project I have on monday and trialing it here saves me loads of troubleshooting on site. My 1608i IP phone is connected to the London PBX and working flawlessly. Fortigates, I absolutely love them! So powerful.
Everything on the Prime so far is top notch. ESXi is perfectly stable under any load with my configuration, I have had it running for days at a time at varying loads without so much as an issue. No PSOD, no problems. ESXi sensors in terms of temps and other readings are still an issue with this board that I am yet to properly look at but seen as I have tuned the system in windows I know for example what the cpu is doing temp wise under common loads I am using. In terms of server build environments I can say that so far the prime is certainly a board that at this point just works with current ESXi releases.
Right now I cannot recommend the Taichi due to multiple issues with that board under ESXi, you can get it working and well if you plan on having san based back end storage, for a decent lab/semi production it could work potentially but in a production world its so far away from where it needs to be that I can't possibly recommend it. You will be forever pulling out your hair chasing issues which appear to be more bios/firmware related or could be related to how esxi is reading the controller on that board. I can't explain what is going on but it's not working like it should.
The prime under esxi, I get no error codes. No boot problems, ram compatability is an issue but I am running 64gb over 8 sticks @ 3066 on the Prime much like I was in the Taichi. The prime is a plug and play experience in terms of ease of use, it boots to esxi as quickly as the Taichi so you wouldn't have any issues there. Both of them booted esxi quickly even from a USB, around a minute to be up and running but I can time it if it helps.
So far I have tested just the two boards and for your basic environment the prime offers enough but is missing nvme raid and being the bottom rung board on the asus ladder they don't seem to be supporting it anything like they are with the ROG boards. The Taichi has 3x native M.2 whereas the prime has just 2 and one is poorly positioned imo, Taichi has 8 sata, the prime has 6, Taichi has dual nic's which right now I could really do with but the prime has just 1. To be honest the list goes on and another thing I have noticed while I was fiddling is that the Taichi recovers much better at total loss of power or failed overclock than the Prime does. It also has sensible buttons for bios reset and system reset on the board whereas the prime has a poorly positioned jumper. In fact the layout of the prime in general is just not nearly as nice as the Taichi in almost every respect. If the Taichi worked nearly as well as the prime does under esxi then it would be perfect... Which leads us to the gigabyte lineup.
Gigabyte are about to release a board called the DESIGNAIRE x399, its almost identical in layout to the Taichi and offers pretty much exactly the same feature set. Pricing is looking to be £400 to £450 and it might, just might be the perfect board for an ESXi build if it works without issue. It will cost around £400 to find out but I think I will give it a shot over the next few months. Right now the asus lineup seems to be the go to for esxi boards but don't count on any help at all from them if you have any minor issues as their customer support is appalling.
Put simply if the Prime feature set is enough for your requirements its a sound esxi board.
Everything on the Prime so far is top notch. ESXi is perfectly stable under any load with my configuration, I have had it running for days at a time at varying loads without so much as an issue. No PSOD, no problems. ESXi sensors in terms of temps and other readings are still an issue with this board that I am yet to properly look at but seen as I have tuned the system in windows I know for example what the cpu is doing temp wise under common loads I am using. In terms of server build environments I can say that so far the prime is certainly a board that at this point just works with current ESXi releases.
Right now I cannot recommend the Taichi due to multiple issues with that board under ESXi, you can get it working and well if you plan on having san based back end storage, for a decent lab/semi production it could work potentially but in a production world its so far away from where it needs to be that I can't possibly recommend it. You will be forever pulling out your hair chasing issues which appear to be more bios/firmware related or could be related to how esxi is reading the controller on that board. I can't explain what is going on but it's not working like it should.
The prime under esxi, I get no error codes. No boot problems, ram compatability is an issue but I am running 64gb over 8 sticks @ 3066 on the Prime much like I was in the Taichi. The prime is a plug and play experience in terms of ease of use, it boots to esxi as quickly as the Taichi so you wouldn't have any issues there. Both of them booted esxi quickly even from a USB, around a minute to be up and running but I can time it if it helps.
So far I have tested just the two boards and for your basic environment the prime offers enough but is missing nvme raid and being the bottom rung board on the asus ladder they don't seem to be supporting it anything like they are with the ROG boards. The Taichi has 3x native M.2 whereas the prime has just 2 and one is poorly positioned imo, Taichi has 8 sata, the prime has 6, Taichi has dual nic's which right now I could really do with but the prime has just 1. To be honest the list goes on and another thing I have noticed while I was fiddling is that the Taichi recovers much better at total loss of power or failed overclock than the Prime does. It also has sensible buttons for bios reset and system reset on the board whereas the prime has a poorly positioned jumper. In fact the layout of the prime in general is just not nearly as nice as the Taichi in almost every respect. If the Taichi worked nearly as well as the prime does under esxi then it would be perfect... Which leads us to the gigabyte lineup.
Gigabyte are about to release a board called the DESIGNAIRE x399, its almost identical in layout to the Taichi and offers pretty much exactly the same feature set. Pricing is looking to be £400 to £450 and it might, just might be the perfect board for an ESXi build if it works without issue. It will cost around £400 to find out but I think I will give it a shot over the next few months. Right now the asus lineup seems to be the go to for esxi boards but don't count on any help at all from them if you have any minor issues as their customer support is appalling.
Put simply if the Prime feature set is enough for your requirements its a sound esxi board.
Last edited:
No Problem and I wish you luck with your build.
It looks like AMD had a couple of offers running to shift some threadripper stock so it was really nicely priced last week. I paid full release price in excess of £950 and haven't at all been disappointed with what it has to offer in terms of performance even at that price. It rips through everything I throw at it with relative ease and is working an absolute charm in server environments. Storage wise it can be whatever you want it to be really, it has enough pci-e that you can make an nvme monster if you need the I/O. I would love to be able to make a massive raid 10 array of nvme and see if you could expose that as storage to esxi because that kind of I/O performance was pretty much out of question in anything other than some very expensive storage arrays not that long ago, I am thinking along the lines of pure and 3par.
To have that in a single box in a mini form is great and opens up performance in the lab that was out of reach below 10k a year back. Granted if you are learning how to manage and maintain or provision storage out of something like a 3par or EVA then it's about as useful as a chocolate teapot but if you need I/O performance in the lab then it opens up a sensible option even if it takes some fettling to get it working.