1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Home Lab Threadripper Build Thread.

Discussion in 'Project Logs' started by Vince, Aug 12, 2017.

Tags:
  1. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    Just to say that I will give you an update on this but it might take me another week or so. Although I failed miserably with the linux piece on IOMMU I hopefully won't fail so badly with this. To be honest I still have no idea how to do what I needed to do in linux and how anybody likes linux is still totally beyond me and in 15 years I have come across more unix machines than linux boxes in businesses. That being said had I spent a lot of time in linux I might be able to easier solve some of these issues.
     
  2. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    Little update. That fella posted an updated video so I have purchased an Asus board to test which will be with me tomorrow. Very happy in this Instance to be proved wrong and am glad that there are solutions out there that just work, this Taichi had me totally convinced that the problem ran deeper than the hardware but I have been wrong before and it looks like I am again. I went for the prime and if that's no good it will go back and I'll work my way up the stack until I get to the rog.

    Now I just have to deal with the lack of stuff on the board I have ordered. It will be here tomorrow so I guess we know if the prime is an option in a day or so. Dunno what to do with my couple of month old Taichi but may just use this as an excuse to build another TR machine for the wife.
     
    Last edited: Oct 11, 2017
  3. Aneskimo

    Wise Guy

    Joined: Jul 10, 2013

    Posts: 1,023

    Location: Space

    A bit deflating but at least there is a solution, also if you can get away with another build with the missus then why not? Mine would kill me :D
     
  4. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    There is indeed but what I would find interesting is if the prime has issues that the rog doesn't. I would be amazed if the only two boards that are marketed more towards the professional market are not up to scratch with the ROG when it comes to using them for more professional type applications. I mean this is for professional use more than anything and boards with gaming focus, gaming in the title etc are not really something I want to be buying if i'm doing it through the company, which was the case with the Taichi even if it isn't with the Prime. All the other stuff i'm now having to buy to get my lab up and running 100% is just a tad annoying but I guess if you want to be on the bleeding edge then you have to suck it up.

    One issue I do have apart from outrageous prices, which is superficial to say the least is that I don't like the look of the rog at all really and that includes the strix. It just does nothing for me in terms of aesthetics where as the prime seems much more clean. I just hope it works.
     
  5. Mercutio

    Mobster

    Joined: Jun 11, 2003

    Posts: 4,868

    Location: Sheffield, UK

    Subbed.

    There's a 4 node RAC 12c oracle build to throw at a suitably endowed ESXi setup on threadripper one day.

    BTW - did you ever get further with tuning the ram or still 3066? I'd want it to be home-lab and next to no compromises on the gaming front (tall order, I know :p ).
     
  6. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    Ram which is 3466 will do 3200 in the Taichi with cas 14 timings and 1.1 soc voltage, it was stable but felt like I was trying to squeeze every last bit out of it to get it there, it wasn't doing much for temps either so I did go back to 3066. To be honest others have had a more plug and play experience with the same ram in other boards so later today I should be able to compare how the Prime fares. Gotta be honest the last thing I wanted to do was tear down the whole build to replace the board but tonight that is what will happen.

    In terms of gaming I can't say you would really be compromising with memory at 3000+, no game out there today really touches a 1950x threadripper cpu for more than about 10% so to some extent you are constrained to its single core performance and while it's not quite up there with the 7700k and 8700k in terms of gaming performance it's certainly no slouch, you do see gains at 3200 there is no doubt but they get smaller and smaller as you go past 3000. I have my 1950X paired with a vega 56 (running 64 air bios, overclocked and undervolted to 1680/1100) and so far for the majority of games I just stick it at 4k with the majority of the eye candy turned up and away she goes. Project cars 2, 4k with all the bells and whistles sits nicely pinned at pretty much 60fps, PUBG I have to reign it back a bit to 2k with medium settings and ultra draw and I sit somewhere around the 80 to 100 frames. I have installed a few games and it does what you would expect it to do, goes more than fast enough with masses left in the tank.
     
  7. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    Couple of pics of the swap over:

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]

    Bitter sweet for me as the ASRock is the nicer board but the prime installed esxi 6.5 update 1 rollup straight off the bat. I can see all my devices, connect to my nfs datastores and importantly sata works and I can use sata devices as datastores.
     
    Last edited: May 4, 2018
  8. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    [​IMG]

    [​IMG]

    [​IMG]

    [​IMG]
     
    Last edited: May 4, 2018
  9. mcanela

    Associate

    Joined: Oct 13, 2017

    Posts: 9

    amazing! no problems at all?
     
  10. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    Nope, not a single issue, 100% plug and play.
     
  11. mcanela

    Associate

    Joined: Oct 13, 2017

    Posts: 9

    Glad to hear that !!!! Congratulations Vince we celebrate your aport
    did you had a guideline whit your experience?

    i´ll keep on eye on the progress ... still i´m waiting the release of EPYC 8C /16T with supermicro mobo for my configuration

    Thanks Vince thanks
     
  12. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    In terms of guidelines I would say, and this is a guess right now until I dig deeper, but if your buying a threadripper board for ESXi look for boards that don't have the full compliment of 8 sata ports. The one common theme seems to be that the boards with 6 ports, so all the asus boards, seem to not have the issue with sata. At first I thought it was down to the number of controllers exposed to esxi as the device ID 0x7901. I don't think however after last nights investigations that this is the case. Both the ASRock Taichi and the Asus Prime expose two devices both with the same ID to esxi, that device is the AMD FCH Sata Controller. In terms of the devices in ESXi I cannot find any difference between the boards so right now I am thinking that there are some firmware/bios issues with at least the ASRock that cannot be resolved with simply poking it.

    I can now push forward with further testing. If you guys want I can do a video or two of, well... whatever you want to see. I am happy to set up a VM with GPU pass through running a game or two? Also happy to just load up the platform and put it under a ton of stress if that's what you want to see? Basically from here on out Ill be guided by you good folks in here. Tell me what you want me to test and ill build it up and put the results in here or in a video on youtube. In terms of what I need to achieve I am pretty much there now, my requirements are met. I am not giving up on the Taichi though and think I am going to build up another threadripper machine which will mainly be for the wife but will also serve as a place to continue testing that board as ASRock/VMWare push out updates.

    So to clarify, and it pains me to say it because I really liked the ASRock but right now its just a no go for ESXi. What I can say for sure is that the X399-a prime is a very good, relatively low cost candidate when it comes to building a workstation with ESXi in mind.
     
    Last edited: Oct 13, 2017
  13. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    Doh quoted rather than an edit...
     
  14. cccy

    Associate

    Joined: Sep 10, 2017

    Posts: 12

    Glad to see that you eventually got it working!

    By the way, may I request that you try the following:
    1) Setting up a couple VMs and stressing the ESXi host to 100% CPU usage for some time? (Some motherboards boards are having issues with ESXi PSODing when the CPU is pushed hard)
    2) Keeping ESXi running continuously 24/7? (Some motherboards are having issues keeping ESXi running 24/7 and PSOD after a couple hours even when fully idle)
     
  15. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    Not a problem, I will build and clone 5 or 6 windows VM's and have each one running an instance of prime or something, I think an hour or two should give us a decent real life scenario while proving stability?
    In terms of leaving it running ESXi, ill do an extended test overnight and through tomorrow and see how it goes.
     
  16. SupraWez

    Wise Guy

    Joined: Nov 17, 2007

    Posts: 2,161

    Top man for testing this, saves others going through the same issues.
     
  17. mcanela

    Associate

    Joined: Oct 13, 2017

    Posts: 9

    after the stability stress test will be nice a:
    GPU and audio chipset passtrough to a win10 VM with a Gaming stress test :)

    Sata ports passtrough to a Freenas VM

    Regards
     
  18. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    Cheers lads. Feels good that it is now running, so far so good on overall stability, its been on now for a couple of hours. I forgot to leave it overnight but have been starting to build up a little test environment, so far we have a 2012 server, a 2016 server, one windows 10 machine. 8 cores on the 2012 server have been running prime now for about an hour and everything is ticking along nicely.

    I'm going to build up another 2008r2 server then I will stress it all some more :)

    [​IMG]

    These arrived:

    [​IMG]

    Not sure if I actually need them anymore, also they could be faster so not sure if I am going to keep them for this build. Might put them in the ASRock board.
     
    Last edited: May 4, 2018
  19. cccy

    Associate

    Joined: Sep 10, 2017

    Posts: 12

    Looks like its going pretty nicely thus far! Do keep us updated on how the stress tests went!
     
  20. Vince

    Man of Honour

    Joined: Oct 30, 2003

    Posts: 9,186

    Location: Essex

    Stress test for several hours at load and 24 hour (almost) went absolutely fine. The board appears to be stable. I am not sure it is the board for me just yet as I have seen the board in the link below is about to drop, also asus have some of the worst customer service I have ever had the misfortune of having to use:

    https://www.gigabyte.com/Motherboard/X399-DESIGNARE-EX-rev-10#kf

    Am very tempted to buy that one and give it a go. Feature wise etc it is exactly what I am looking for. Did notice a small issue building a windows 10 machine which I am looking into now. Particularly that it wouldn't accept more than 2 cores but it could be something to do with how I built it so am going back over the win 10 gpu passthrough later on this week.