Home Lab Threadripper Build Thread.

[Vince]

Glad to see that you eventually got it working!

By the way, may I request that you try the following:
1) Setting up a couple VMs and stressing the ESXi host to 100% CPU usage for some time? (Some motherboards boards are having issues with ESXi PSODing when the CPU is pushed hard)
2) Keeping ESXi running continuously 24/7? (Some motherboards are having issues keeping ESXi running 24/7 and PSOD after a couple hours even when fully idle)

Not a problem, I will build and clone 5 or 6 windows VM's and have each one running an instance of prime or something, I think an hour or two should give us a decent real life scenario while proving stability?
In terms of leaving it running ESXi, ill do an extended test overnight and through tomorrow and see how it goes.

 

[Vince]

Cheers lads. Feels good that it is now running, so far so good on overall stability, its been on now for a couple of hours. I forgot to leave it overnight but have been starting to build up a little test environment, so far we have a 2012 server, a 2016 server, one windows 10 machine. 8 cores on the 2012 server have been running prime now for about an hour and everything is ticking along nicely.

I'm going to build up another 2008r2 server then I will stress it all some more



These arrived:



Not sure if I actually need them anymore, also they could be faster so not sure if I am going to keep them for this build. Might put them in the ASRock board.

 

[Vince]

Looks like its going pretty nicely thus far! Do keep us updated on how the stress tests went!

Stress test for several hours at load and 24 hour (almost) went absolutely fine. The board appears to be stable. I am not sure it is the board for me just yet as I have seen the board in the link below is about to drop, also asus have some of the worst customer service I have ever had the misfortune of having to use:

https://www.gigabyte.com/Motherboard/X399-DESIGNARE-EX-rev-10#kf

Am very tempted to buy that one and give it a go. Feature wise etc it is exactly what I am looking for. Did notice a small issue building a windows 10 machine which I am looking into now. Particularly that it wouldn't accept more than 2 cores but it could be something to do with how I built it so am going back over the win 10 gpu passthrough later on this week.

 

[Vince]

I setup ESXi successfully on my Zenith as well. VGA passthrough is working fine, however, I can't get any USB devices recognized in the VM. So atm can only control through the ESXi console. Do I have to get PCI-e expansion card for USB then pass through that?

Shouldn't have to. USB on my board seems to just work. Once I have the VM up and running (with the usb controller added) in esxi I just mount the USB device by firing up the VM and then right clicking on the USB device right at the bottom of the console window and pressing connect. USB devices are then available through the VM. Can do a little video on this or a couple of screen shots if it helps?

 

[Vince]

Hi Vince.. Tell me about the temperatures... Did the AIo corsair manage it well?

Saludos!

Funny you should mention this. Temps etc are not available in esxi because of lack of support from the board. Ill have to get some screen shots up of this. I only know that it was fine as I have loaded it up for hours on end in windows and stressed it there. For example, I know that the temps with all cores under prime in windows reaches no more than about 70 degrees. I can only assume that we were hitting the same kind of temps in the VM stress tests. I will update later this evening though with some details.

 

[Vince]

That Gigabyte board does look good, though rather than getting 3 motherboards deep in TR, maybe wait and go Epyc instead!

I am actually fairly happy with throwing the money at the boards, worst case scenario is that I sell them on after I have had my wicked way with them. EPYC is tempting but at the end of it all it's not going to offer me any more (at least in the lab) than TR, TR perfectly suits my requirements in terms of core count and grunt so really for me, apart from the compatibility side of things, TR is perfectly adequate.

 

[Vince]

Nice board, seems to be marketed towards gaming and VR which TR isnt aimed at, got some nice features though, wonder what vmware compatibility is like.

It's the features that I like, 8 sata ports (warning signs here), dual lan, 3x m.2, more pci-e than the 399-A. Just seems well rounded and much like the Taichi feels like pretty much the perfect candidate for a lab. The 399-A falls well short as its lacking in pretty much all areas compared to not only the Taichi but also all other x399 boards. Watch this space

If we have to go through every x399 board out there, we will, I will find the perfect lab board.

 

[Vince]

If you could post a few screen shots that would be helpful, thanks. I didn't see any options for this at the bottom of my console window but perhaps I'm overlooking something. I did add the controllers in my VM settings though.

I will post up some stuff later. Can't say for sure if it will be tonight as I need to tear the machine down again to get my gfx AIO properly seated. Spent hours last night trying to get the Vega card under an AIO and gave up really late with frankly not great results. Try again tonight I think.

 

[Vince]

Fair enough, but it's vmware compatibility that made me suggest it, I have a TR system as my VMHost - but luckily I'm not limited to VMWare.

Just out of interest - what hardware are you planning on adding? I'd be very curious to know what combination of bits would exceed even the most basic X399 mobo's! I have a proper dual port Intel NIC, a SAS HBA, and GTX 1070, still leaves a x16, x8 and a x1 free at minimum (not at home to check!).

m.2 nvme drives, lots of them... the Prime has 2 slots, doesn't at this point support nvme raid and to be honest I just don't really like the board or Asus, yes its functional and works but it really is bottom of the barrel in terms of x399. When the Asus Fury Quad hits ill no doubt be adding more nvme than I really need. Put simply though I want a board more in line with the Taichi, more PCI-e, more sata, more M.2, More Nics, better layout. I didn't buy TR to be board limited yet that right now is how I feel.

 

[Vince]

Edit done it again, quote instead of edit... Did the edit button used to be the other side on the old forum software?

 

[Vince]

I setup ESXi successfully on my Zenith as well. VGA passthrough is working fine, however, I can't get any USB devices recognized in the VM. So atm can only control through the ESXi console. Do I have to get PCI-e expansion card for USB then pass through that?

Your right. You can do it that way in workstation and if I remember rightly you can also do it this way in the vsphere 5.5 client. Console in this 6.5 doesn't have the option. Why they can't just let us use a decent vsphere client is beyond me.

I'm done for tonight after fighting with and finally conceding to a gpu aio but will update as I did pass through a usb key to a vm when I was building that Windows 10 machine a few days ago I just don't remember exactly how. I will try and fire it all up again tomorrow night and update this thread with some more results.

 

[Vince]

Hello Gents, Been a while as I have been pretty busy with all kinds, mot's on cars, issues with BT that seem to have been dragging out forever, you know those issues where you make loads of progress then seem to take several steps back? That's basically the sum of my last two weeks. I have made quite a bit of progress in setting up my lab though, the fortigate is all installed and I have configured policy based VPN's with multiple phases between home and the London office, something that I needed to get done for a small project I have on monday and trialing it here saves me loads of troubleshooting on site. My 1608i IP phone is connected to the London PBX and working flawlessly. Fortigates, I absolutely love them! So powerful.

Hi Vince! any news about your progress?

Saludos

Everything on the Prime so far is top notch. ESXi is perfectly stable under any load with my configuration, I have had it running for days at a time at varying loads without so much as an issue. No PSOD, no problems. ESXi sensors in terms of temps and other readings are still an issue with this board that I am yet to properly look at but seen as I have tuned the system in windows I know for example what the cpu is doing temp wise under common loads I am using. In terms of server build environments I can say that so far the prime is certainly a board that at this point just works with current ESXi releases.

Hello Vince, joined for you. I replied to the VMWare support but I think it's much better here.

Thank you for your really hard work, it's a time saver ! We are looking to test a build with ESXI too with Threadripper. Nothing is purchased yet, but we were looking for the Taichi board, however it seems it's a bit complicated and might be an issue for future ESXI updates.

With the Prime board, what are the disadvantages ? Only the missing features ? I've read about issues with error code AD, and post being way too long. Is the ASrock faster to boot ? What about the Gigabytes ? We are interested in the practical experience, ease of use, no lag at boot, easy BIOS, good RAM compatibility.

Right now I cannot recommend the Taichi due to multiple issues with that board under ESXi, you can get it working and well if you plan on having san based back end storage, for a decent lab/semi production it could work potentially but in a production world its so far away from where it needs to be that I can't possibly recommend it. You will be forever pulling out your hair chasing issues which appear to be more bios/firmware related or could be related to how esxi is reading the controller on that board. I can't explain what is going on but it's not working like it should.

The prime under esxi, I get no error codes. No boot problems, ram compatability is an issue but I am running 64gb over 8 sticks @ 3066 on the Prime much like I was in the Taichi. The prime is a plug and play experience in terms of ease of use, it boots to esxi as quickly as the Taichi so you wouldn't have any issues there. Both of them booted esxi quickly even from a USB, around a minute to be up and running but I can time it if it helps.

So far I have tested just the two boards and for your basic environment the prime offers enough but is missing nvme raid and being the bottom rung board on the asus ladder they don't seem to be supporting it anything like they are with the ROG boards. The Taichi has 3x native M.2 whereas the prime has just 2 and one is poorly positioned imo, Taichi has 8 sata, the prime has 6, Taichi has dual nic's which right now I could really do with but the prime has just 1. To be honest the list goes on and another thing I have noticed while I was fiddling is that the Taichi recovers much better at total loss of power or failed overclock than the Prime does. It also has sensible buttons for bios reset and system reset on the board whereas the prime has a poorly positioned jumper. In fact the layout of the prime in general is just not nearly as nice as the Taichi in almost every respect. If the Taichi worked nearly as well as the prime does under esxi then it would be perfect... Which leads us to the gigabyte lineup.

Gigabyte are about to release a board called the DESIGNAIRE x399, its almost identical in layout to the Taichi and offers pretty much exactly the same feature set. Pricing is looking to be £400 to £450 and it might, just might be the perfect board for an ESXi build if it works without issue. It will cost around £400 to find out but I think I will give it a shot over the next few months. Right now the asus lineup seems to be the go to for esxi boards but don't count on any help at all from them if you have any minor issues as their customer support is appalling.

Put simply if the Prime feature set is enough for your requirements its a sound esxi board.

 

[Vince]

Thank you!
For now the Asus Prime would do the job. However we are waiting for price drop, possibly after Christmas. In Canada it just increased 200$ since Friday !

No Problem and I wish you luck with your build.

It looks like AMD had a couple of offers running to shift some threadripper stock so it was really nicely priced last week. I paid full release price in excess of £950 and haven't at all been disappointed with what it has to offer in terms of performance even at that price. It rips through everything I throw at it with relative ease and is working an absolute charm in server environments. Storage wise it can be whatever you want it to be really, it has enough pci-e that you can make an nvme monster if you need the I/O. I would love to be able to make a massive raid 10 array of nvme and see if you could expose that as storage to esxi because that kind of I/O performance was pretty much out of question in anything other than some very expensive storage arrays not that long ago, I am thinking along the lines of pure and 3par.

To have that in a single box in a mini form is great and opens up performance in the lab that was out of reach below 10k a year back. Granted if you are learning how to manage and maintain or provision storage out of something like a 3par or EVA then it's about as useful as a chocolate teapot but if you need I/O performance in the lab then it opens up a sensible option even if it takes some fettling to get it working.

 

[Vince]

Word on the street is that the 1.80 BIOS update (which updates AGESA to 1.0.0.4) fixes ESXi on the ASRock motherboards. Can anyone confirm?

I will refit the board tomorrow and test. I hope it does

 

[Vince]

It seems like a lot of folks here are using this thread to find out about the issues of ESXi with Threadripper, so I shall put down my own notes with my ASUS PRIME X399-A:

1) USB device passthrough is a hit and miss on the USB 2.0 ports (Which are only available as internal headers), some USB devices work fine, while others get detected by the VMs but do not function correctly (Unknown if a future ESXi/BIOS update may fix things).
As for the USB 3.0 ports, I have had no success at all (USB devices get detected by the VMs, but do not function correctly).

2) For those who find the lack of network ports to be an issue, I have a HP NC364T network card plugged into the PCI-E x4 slot and it gets detected and used by both the BIOS and ESXi without any issues thus far (Does not require any driver injection on ESXi).

3) I have a AMD R7 260X as my passthrough GPU for one of my VMs, and ESXi refuses start the VM (Giving a passthrough error) if the GPU is plugged into the first (nearest to the CPU) PCI-E x16 slot. It however works fine with the second PCI-E x16 slot.
I have also tested a ATI Radeon HD4550 in both PCI-E x16 slots and had no success in either slot.

These are interesting results. I haven't yet had any issues with usb or gpu passthrough. I tried both an RX480 and Vega and managed to boot VM's with the gpu passed through in both cases. As for usb devices I was mainly passing through USB storage devices which were mainly on case USB's via headers. But I also passed through my Thrustmaster t500rs as a random device but don't know if this was plugged into USB 2 or USB 3.

Keep us updated though as its interesting and helpful. What usb devices are you having issues with? I wonder if I have any similar that I could test?

Gents I was going to rebuild my rig in the Tiachi today, I went out bought TIM and all that good stuff but am feeling very fragile today after getting smashed last night with the wife (wedding anniversary) and really don't feel up to ripping it all apart today. Perhaps ill give it a go in the week.

 

[Vince]

Congrats on the anniversary!

Cheers buddy. Time flies as they say I think tomorrow might be the day. Swap out the board and test the Tiachi, if it works i'll be straight in with the water cooling and then it's job done

 

[Vince]

I can confirm that the asrock Fatal1ty X399 Professional Gaming bios 1.80 works with VMware vSphere Hypervisor ESXi 6.0.0.U3a-5572656.x86_64. I have not done any real testing as yet though. Both Intel nics are present and esxi detected my 1tb Nvme ssd as well. I took a gamble on this motherboard and I am glad how its working out so far.

Don't suppose you have any sata drives in there? I managed to get all but sata working, my nvme etc worked without issue. The real problem was getting sata exposed to esxi.

 

[Vince]

300 CAD$ drop on the 1950x on Monday, purchase done with the Asus Prime. Just saw the posts on the BIOS 1.8, too bad. At least the Asus one was a bit cheaper. I'm just bothered by the eATX form.

I will not test any RAID, this is because this machine will possibly be used for users, with multiple virtual machines as desktop, connected from clients still running on old hardware. This was a way for my boss to skip the hardware upgrade of each computer here. At 2500$, it's much cheaper than replacing 10 computers. However the current Esxi build we have is running on Intel at low frequency and our applications are single core, so we are wasting time waiting for the applications to start or process each request. The Threadripper is a perfect replacement, high frequency and cheap price.

I wouldn't worry about the eATX it is literally 2cm wider than the ASRock, not really worth the eATX tag to be honest.

 

[Vince]

I did some further testing and it appears I can pass through video cards and sata control. I was able to select the devices for pass through, rebooted the host and the devices were ready to be assigned to a VM. But once assigned to a VM, the VM would not boot. If I removed the pass through video card, the VM boot up fine. I also loaded up the lastest Esxi 6.5 and same issue.

I did some further testing and it appears I can’t pass through my video cards and sata controller. I was able to select the devices for pass through, rebooted the host and the devices were available and ready to be assigned to a VM. But once assigned to a VM, the VM would not boot. If I removed the video card I was passing through, the VM boot up fine. I also loaded up the latest ESXi 6.5.0.U1-5969303.x86_64 and same issue.

On a more interesting thought. My old setup is a Xeon e5-2696 V3 with 128 GB RDIMM. I was planning on migration over to the thread ripper 1950x due to its higher base clock vs 2696 v3. What was interesting was that I did some synthetic bench marks with one of my VMs that I assigned 4 vcores. I then copied and migrated that same VM over to the thread ripper host. Used Cinebench to conduct the benchmarks. I have to say, I was surprised by the results. In all the tests, the Xeon e5-2696 kept coming out on top. The thread ripper scored 456 and the Xeon e5-2696 v3 scored 495.

Given the pass through issues with the thread ripper build I have decided to stay with my current setup for now.

I can help you get the gpu pass through working as there are some modifications you will need to make to some esxi files to achieve this without faffing around.

Sounds like the ASRock might still have some ahci issues but can't confirm as I'm yet to put my board back in. Probably a job for the weekend now.

Performance wise I am surprised, out of interest what ram and how fast are you running it on the 1950? Also if you could throw me the files I could try and help confirm your results and compare against the Asus board before I rebiluild.

 

[Vince]

If I have to do that I'll just buy ASUS ...
Think is that all other boards work fine, but ASRock managed to screw this one up ... We need to force them to fix it ... not just add HBA ...

I have pulled vmkernel log and it shows vmw_ahci probing ports and it actually finds SATA HDDs ... 3 in my case ...

after detection I get this:

2017-11-27T19:15:43.227Z cpu1:65971)PCI: 88: Device 0x384e4302e1e5ddc2 is not an PCI vmkDevice
2017-11-27T19:15:43.227Z cpu1:65971)Device: 2482: Module 0 did not claim device 0x384e4302e1e5ddc2.
2017-11-27T19:15:43.228Z cpu1:65971)PCI: 88: Device 0x124f4302e1e5de6f is not an PCI vmkDevice
2017-11-27T19:15:43.228Z cpu1:65971)Device: 2482: Module 0 did not claim device 0x124f4302e1e5de6f.
2017-11-27T19:15:43.228Z cpu1:65971)PCI: 88: Device 0x3234302e1e5e2fe is not an PCI vmkDevice
2017-11-27T19:15:43.228Z cpu1:65971)Device: 2482: Module 0 did not claim device 0x3234302e1e5e2fe.

Looks to me like there is still something going on with the taich, I am not even sure that given this evidence I will bother putting it back in until we know it is fixed. I have spent hours messing around with it and have conceded that I still have literally no idea why it doesn't work. I still have the Prime in at the moment but I do keep looking at other boards as I want to settle on one so that I can water cool it all. I keep deliberating over the designare but still haven't pulled the trigger as lack of u.2 and untested esxi credentials are kinda putting me off as I could just end up with yet another x399 board that doesn't work with esxi.

Truth be told I keep thinking that I might just bite the bullet and buy the rog after Christmas. At least then I wont be tempted to buy another board as it has probably the best mix of features even if asus customer support sucks, it's still the only fully featured board (apart from the single lan port) that we know just works. In my opinion that lack of 2 sata ports is easily made up for with the dim.2 and 3x m.2 slots. If I sell the Tiachi and the Prime I might get some of the money back and then I could grab me a rog.

If only somebody could confirm that the designare works then I would buy one straight away but given xmas is just around the corner I think the mrs will kill me if i buy yet another board.

For what it is worth I posted a link to this thread and requested help in the ASRock motherboard thread which the vendors apparently visit but as of yet nothing.