Home Lab Threadripper Build Thread.

Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
Ok guys I'm up and running with the prime. One question, I got my RAM at 3200 CL16 but how to I make it work @ CL14 without adjusting everything manually ?

Edit:
OK found it, it's not called XMP, it's called DOCP

It's still imo a huge disappointment that we still only know of the ASUS boards working. I am saving up the pennies now to buy the Zeneth as I need to settle on a board before I put all my water cooling gear in and I am not sure I can be a test bed for yet another board even though I do really like the DESIGNARE.

Good to see you up and running though and the Prime really does just sail through. I am convinced it's something to do with the number of sata ports, there must be a reason that the top end ROG only has 6 and the boards that appear not to work correctly have 8. It's also why I can't justify another 350 quid on yet another board to find out it's not working again.
 
Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
Sorry, for some reason I didnt get notifications on this thread... so am I right that the Asus boards are fully functional with ESXi, inc VGA passthrough, inc USB HOST (not device) passthru and SATA? I may just have to learn ESXi now with my old kit before splurging on a 1950X setup....

EDIT: Reason to passthru a USB controller is to allow hotswap. If you pass through a Mouse for example you cannot unplug and plug in a USB stick and have it detected can you?

All of these things work for me on the Asus board.
 
Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
I successfully tested ESXI within a VM Workstation on my Windows installation. ESXI was installed on a USB key.

But when I tried to boot on this key, or install a fresh ESXI on this key, I got a pink screen of death ?? fcpu 15 invalid id ? something like that.

Any idea ? I'll look on google of course but you might already know if I have a bad setting in the Prime motherboard.

Edit:
The exact message is "pcpu 15 has a bad apicID 0xf"

I searched on google and found out it's an issue with ESXI 6.5 and Ryzen. Disabling SMT gave the same message but 8 instead of 15.

The issue is your vmware version. Versions before 6.5 update 1 had a cpu core limit of 15. Even with smt disabled assuming your on a 1950x your still going to have 16 cores. At least this is my first gues. If you post the full Iso name that would be helpful.

Happy to post screen shots tomorrow of all the bios settings you need enabled or that are helpful when virtuaising. Off the top of my head you only need to enable cpu virtualisation to be able to pass the installer, if i remember rightly it took me a while to find in the prime bios.
 
Last edited:
Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
I don't have access to this ISO on the website download page, where to get it ?

Edit:
I searched with the name on google and got it on another download section of vmware website. However I'm not allowed to download it. We have a license at work but for ESXI 5.5. We want to test the machine before purchasing everything.

Edit2:
Found the iso somewhere else, hope it's OK. How do I check if it fits the hash ?

Right RDP'd into my home machine to look at my nas, the highlighted iso is the fella that works flawless every time:

 
Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
I just spotted this in New posts and was like... I recognise that thread :)

I recently put the Taichi back in my machine and moved the prime over to another machine running TR-JR (1900x).

Im thinking it might be time to re roll this and see if the Taichi works yet?
 
Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
Hi Vince and OCUK Community.

I've been following this thread for a while and reviewing everything. I am currently attempting to also setup a ESXi lab on my machine. Currently my specs are the following:

Asus X399 Prime
1900x Threadripper 8C /16T
16 GB RAM DDR4 3200 MHz CL16
GTX1080
NVMe 500GB Samsung 960 Evo
1000w PSU Corsair RM1000

Besides SVM, are there any other specific configurations that I need to change in the motherboard to get the pass-through to work? I've been trying to follow a lot of details in this thread, as well as details in the reddit page https://www.reddit.com/r/Amd/comments/72ula0/tr1950x_gtx_1060_passthrough_with_esxi/, but i'm currently a bit stuck.

Any details about the optional MB configurations that can be used for this task?

Thanks in advance for any replies.

There are some changes you need to make to some vm config files for your pass through to work correctly. I will try find you a link :)
 
Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
great case choice

Thanks dude... I should really wack some updates out on this. I am sure the world wants to know if ASRock ever sorted the Taichi for ESXi, even I want to know! I put the board back in and everything just to find out!

I still haven't tried mind. I ended up building a second TR rig for ESXi! Over this weekend I am making it my mission to answer the question of does it work. Fwiw I did end up adding a shed load of GPU's into the esxi lab and testing gpu passthrough on the x399-a. I had passthrough running on Vega, RX570, RX560, RX550 & RX480. So can confirm that this does work but there are some esxi config files that need tweaking to make it seamless.
 
Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
Can also confirm 6.7 works on my Taichi :) Big thumbs up to the lads at vmware / asrock for finally getting this sorted! I haven't had much in the way of esxi work recently but when my next bit comes along it's gonna be really nice not having to swap boards in and out.
 
Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
If ESXi is working properly then all of those things should already be monitored, certainly on HP hardware you can see all that. If you load the extra HP packages you can also see storage controller status including raid and batteries etc.

This is true but the monitoring is not fully supported outside of pretty much anything that isn't oem hardware. I have never had an unsupported board that properly reports on the hardware in ESXi. Fully working in respect of an unsupported lab is everything apart from monitoring imo. so long as sata and all the other hardware is available to the environment then for a lab that should be enough, in production really id be sticking with HP DL385 gen 10 etc and enough of them for HA and DRS, I mean that's just sensible if you are running in anything which would be classed as medium or above in terms of business size, I was going to buy the gen 10's this year but with 7nm epyc so close i'm now waiting on that. You can bet all the money that when hp release those new 385's that ill be all over them :)
 
Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
I thought about that. Not the best option, so I'm looking at what I can find first.

Somebody should be able to 3d print one, this is the same part that didn't come in my 399 prime. Asus were totally usless. I got one on eBay in the end and now no longer own the board as I gave it to a friend and went back to the Taichi.
 
Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
At work my boss decided to increase the capacity of the server by adding another 64GB of RAM (total 128). Now we can't boot with RAM at 3200, it reverts back to 2133 MHz. This is GSkill 3200 MHz CL14 F4-3200C14Q-64GTZ. In order to be able to run at 3200, we need to set the EFI to be D.O.C.P. It's working fine with 64 or 96GB (and it's also fine at home with my 32GB kit same model).

I tried to increase the cas latency to 16 in manual mode, no good. We've left the D.O.C.P. activated but set the target frequency at 3066 MHz, now it works. I suppose it's running at cas latency 14, but I don't know how to confirm that in ESXI. I've read a few things on google but it's not that clear.

128gb @ 3200 is too much for the imc to take. Remember TR as you up the capacity and banks consumed, that it only properly supports 2133. Read your motherboard manual, mine makes it quite clear on limitations when all banks are filled. I can get all banks running at 2933 but much more than that and it doesn't like it. Reduce the capacity and more is achievable.
 
Man of Honour
OP
Joined
30 Oct 2003
Posts
13,271
Location
Essex
Hello guys, just registered to post here specifically. First of all, many thanks to all of you for putting in so much work in testing and diagnosing X399 boards with ESXi. I'm just about to buy parts for my own brand new server based on this platform. What's the current state of things? Is the Taichi still the recommended and most compatible board, as this thread seems to suggest in the end? What are your experiences running your setups over the long term? I'm mostly interested in the stability of running Windows/Linux/BSD as guests with various passthroughs (GPU, dedicated SATA card, M.2 drives and USB).

The current state of play with the taichi is pretty good. Things appear to keep improving as ESXi release new builds and in it's current state it's workable in the lab for sure, it appears some people are having some issues here and there passing through certain devices etc but personally for me and what I need it for it's been perfect. It's my favourite x399 board of them all! It might not be the best overclocker or the best at everything but it is just a really nice, well thought out and well rounded board. Oh and welcome to my thread! I am sure you will find that the hospitality here is top notch ;)
 
Back
Top Bottom