ASUS Z10PE-D8 WS

Associate
Joined
8 Oct 2008
Posts
1,272
ASUS Z10PE-D8 WS Dual-Socket Haswell Xeon V3 Motherboard

hdj4Z66.jpg
Supposedly has very limited over-clocking options for dual xeons - we shall see.

Found a review
http://www.tweaktown.com/reviews/6866/asus-z10pe-d8-ws-dual-cpu-intel-c612-workstation-motherboard-review/index.html
Scores 99% in every area, not sure I'd give it that high but it is the best motherboard I have ever used - in combination with the chips that is. I think the reviewer is a bit excited :D

gU71aGP.jpg

:eek: So I did the dirty deed :D. - wish me luck.
 
Last edited:
Associate
OP
Joined
8 Oct 2008
Posts
1,272
OCUK did say they hope to be getting some, when... They do not know. Most retailers appear to keep getting the stock pushed back a few weeks.
Gibbo did also mention that he is awaiting a board from asus to do some testing too.

My CPU's have been delayed.. >< So nothing to report yet.
 
Associate
OP
Joined
8 Oct 2008
Posts
1,272
I hope your having better luck than me. - Couriers have *misplaced* my cpus :(
All is insured, however it takes damn ages to sort. - can't see me up and running for a good few weeks at the earliest.


That ram is supposed to work from info I have gathered. 8gb chips are too small :D - need at least the 16gb dimms surely.
 
Associate
OP
Joined
8 Oct 2008
Posts
1,272
I gave up and went to the couriers depo and spent the day arguing... Took 6 hrs.:mad:
Persistence pays - I got my processors :D. I am however mentally and physically exhausted. - That kinda nonsense is really draining...:(

I had to drill a few extra stand off holes for the motherboard - as 3 of the cases existing mounts were not aligned for ssi-eeb boards. Coolers and ram went on ok. I want'd some Noctua NH-D15 's however it looks like all the big coolers either block a ram slot or touch/obscure the first pci slot.- also the weight of 2 of them maybe a issue.. Cable management will have to wait for another day.

The system is now booting up - after a scary couple of hours of trying to decipher error codes and strange beeps. I flashed to the latest bios - thankfully which you can do with the system turned off. It lives ! :D


The bios is pretty comprehensive, lots of new terms and options which I have no idea what they do nor can find any explanation, not sure I'm brave enough to fiddle with them yet.

Overclocking options - are limited, as we know xeons are multiplier locked.
However there are options for manual/auto, AI tune, memory oc'ing various power states etc.

Post times are not fast - I guess the 100 and 1 things that the board has/does, not unusual for a workstation/server board either.

I'll get some screenshots, and maybe a photo or 2 up tomorrow, hopefully a benchmark or 2.
 
Associate
OP
Joined
8 Oct 2008
Posts
1,272
- Me also sorry.

I did call a few times over the last month.. - You guys didn't know when you'd be getting them. - Maybe try to get some good deals on ram, - OCUK currently have the worst selection of all the etailers ;)


I ran a quick bench, with no chip set or any drivers installed, - default everything in bios.

I'll see how I get on improving that. I recon 40k+ is achievable. :eek:

HDwLrIZl.jpg Messy Messy..;)
 
Last edited:
Associate
OP
Joined
8 Oct 2008
Posts
1,272
Huzzah :eek:

jWxeHPr.jpg

As a wise man once said. (May have been 8pack) Over clocking when you have no idea what you are doing is a dangerous thing.

Chips running at 1.073 - 0.964v - hard to tell as the benchmarks completes so fast you can hardly get a look in. temps in mid 40's. - however I'm sure they will get hotter over extended periods of time.

It's a world record score according to Hwbot - can't complain with that. ( although I do appreciate it's due to lack of people with such hardware, nor can I validate as I'm on a incompatible windows version :D... -
 
Associate
OP
Joined
8 Oct 2008
Posts
1,272
Hahah - zx - I fear to know what kinda hardware you are running.

The coolers are two cooler master nepton 280L's. The reason is that the pipes are longer on these so they could go upstairs.
I'd preferred to have gone proper water-cooled, however by the time I'd speccd the bits - that was another £500+. And with the 280L's beating all the aio's and air coolers, it made sense.
 
Associate
OP
Joined
8 Oct 2008
Posts
1,272
These pic's should help explain what I'm doing a bit better.

The poor-man's Caselabs imitation. :p However at 1/5th of the cost - the coolermaster does a fine job. Not sure if to stick more drive caddy's below. Or a low power itx system/nas type system for inception giggles.

eLCHMJe.jpg

And from above - 1 280 rad each side. Currently airflow is going form left to right so one rad's fans is pushing the other pulling. May change the fan configuration around, however it's quite and works at the moment. Well I assume it's quiet, I can't hear anything over the damn quadro whirring away.

RiOSQ7C.jpg -
 
Associate
OP
Joined
8 Oct 2008
Posts
1,272
A little more toying this morning. - Not touched any voltages etc.

Gd24Kz7.png

My theory is that will all the time saved from the faster processing power - I can afford to play a bit ;)
 
Associate
OP
Joined
8 Oct 2008
Posts
1,272
I managed to find two 14 core V3's, now all I need is some ram and another cpu waterblock.
:) Exciting times ahead colin !
Looking at your build log, I'm sure the system is going to be a thing of beauty too. - What case will you be using ?
I had my eyes on the phanteks entho primo, however it appeared to be too limited with rad placements etc when using aio's. The cm stacker prooved to be awesome space wise with it's easy expandability, similar to a caselabs Magnum STH10 (discounting looks, quality, awesomeness etc :D)

I purchased a 64 GB Crucial Kit 4*16GB CT4K16G4RFD4213 1.2V RDIMM. It is identified as the same micron memory on the QVL. (micron make the memory for crucial)
I'll probably get another kit in the coming weeks, however ddr4 is stupidly expensive at the moment. around £550 for 64gb. COME ON OCUK !!

Ideally I'd have gone for the 32gb chips, which there are some samsung ones out in the wild, however by the time I managed to get my card out etc, they are sold. Plus now I have forgotten the product part, so am struggling to find them again :p. - maybe worth a search, if you find them please say.

One thing to note, as I only currently have 4 dimm's - 2 per cpu it appears to actually be running in dual channel mode. - Looks like you have to populate all 8 slots (4 per cpu) to get the full quad channel benefit when running dual cpu's.
I'm Not sure if quad channel memory will make much of a difference for what I am doing. - I guess every little helps, especially in certain workloads.

DDR4 Ram even the best desktop stuff is still very much in it's infancy and basically sucks in comparison to higher end DDR3, transfer speed is faster however the latency/timings are very slow. I am sure this will improve over time.

I have even started to look into ways that can use SSD tech as RAM, The things that I wish todo ideally need 1tb+ of ram (not going to happen with my budget) As the applications mostly use the ram as a holding space for the calculations, not a huge amount of data (well 1tb is huge, but relatively speaking) is actually swapped around so in my mind it kind of negates some of the need for the performance ram actually gives.

It's a given that there would be a performance penalty - even if it is significant. At least the calculations could take place, currently they just cannot be performed as the dataset is too huge, unless spending £20k+ on ram (not even a remote option) :( :eek:

There are some products available such as http://www.sandisk.co.uk/enterprise/ulltradimm-ssd/ which comprise a ssd onboard the dimm, however It looks like they are used only as a fast ssd interface, and not to increase the ram capacity.

Basically I actually want a reverse ram disk - maybe using something like a intel p7300, which is designed for massive throughput and extremely high amount of writes, even if the lifespan was reduced it would be massively cheaper than using actual ram. Just figuring out how todo such a thing is the trouble. I fear I maybe barking up the wrong tree, or looking at the wrong terminology.

On a brighter note what took over 24hrs on my 2600k now only takes 3hrs :eek::cool:
 
Last edited:
Associate
OP
Joined
8 Oct 2008
Posts
1,272
I can't mention names, however look at some price comparison sites, - some reputable retailers have stock at the ballpark I mentioned - besides ocuk don't currently sell any ddr4 ecc (sorry, thats getting old :) *hint*)
8*8gb does appear to be more expensive than 4*16 however significantly less than 8*16.

The stacker needed 3 extra mounting holes drilled (only took 15 min) for the motherboard mounts - the row down the middle. I mounted the board carefully without them, then stuck a sharp pencil through the missing mounting holes to mark the motherboard tray, removed the board, and drilled 3 tiny holes a bit smaller than the standoffs. Then using a bit more muscle I screwed the standoffs in. as they are brass, they threaded the tray on the way in. All worked perfectly. (used a hoover and then magnet, to ensure all particles were gone)

ZX spectrum I think has a lot more experience with air coolers, so I'm sure he can recommend something suitable. - As the ram is 1.2v and doesn't come in massive OC'd flavours, they are low profile without heatsinks/spreaders. so clearance shouldn't be a huge issue. Nor Heat.


Regarding your folding at home etc, a quick read suggests Bigadv Work Units can consume upto 750mb per core ( assume that's what your after, as they net great returns), so depending if hyper-threading is used, 64gb maybe required anyway. I also note that F@H on SMP is a memory bandwidth hog (the more cores the more bandwidth needed) so populating all the slots may make a big difference. Not sure if the increased bandwidth of ddr4 running at dual channel is enough, or quad channel would be better. - of course running 4*16 does give the option for more at a later date.


My 'Grand Plan's' :) haa - I consider them delusions :o
3D graphics & Gigapixel imagery.

I wondered over to the distributed computing subforum.. I fear I'd be pulled into that shenanigans if I dare posted there. ;)
 
Last edited:
Associate
OP
Joined
8 Oct 2008
Posts
1,272
It's been rock solid. Rendering maxed out for the last week.

I really aught to get a UPS - as it's all 1 calculation. I can't find any quiet ones though.

What kind of issues did you have with the X99-E WS ?


What do you guys use to control your case fans when using this board. I'm considering buying a fan controller to use but there is mention of asus utilising its own fan control software. Any help would be great. Thanks
I just let the bios deal with things, - there are various options there. All fan headers are pwm and can be controlled.
 
Last edited:
Associate
OP
Joined
8 Oct 2008
Posts
1,272
Still can't find any propper reviews out yet.

TTL OC3D has a teaser.

He clearly knows something I didn't and is using regular ram for dual cpu's which I didn't think would work. . Maybe it does however and 64gb is the limit ? - those dominators sure do look badass though :)

25slzmh.jpg
 
Last edited:
Associate
OP
Joined
8 Oct 2008
Posts
1,272
From all my previous experience with such things, they wont work.

Only back in the P4 xeon days could you get away running non ECC with SMP.

I'd expect 1 cpu may run with the ram, with a max limit of 64gb. same as the X99-WS boards. Greater amounts of ram or SMP will require ECC, that I believe is a restriction set by intel hardwired into the CPU's.

There are 3rd party ram sinks available (don't look as nice). I'm sure some wise guy will mount the dominator sinks onto ECC. A warning if you try, they are glued onto the dominator ram, quite a few peeps have ripped the chips off the PCB trying to remove them :D.

That said the qvl for ram is very limited, the previous generation motherboards were very finickety, usually minor updates to the bios are required to sort stability for compatibility etc, which won't happen for consumer class ram. Of course luck comes into play.
 
Associate
OP
Joined
8 Oct 2008
Posts
1,272
:) Nice to see you almost have your system up and running Colin.

Venturi - nice post and thankyou for all of the info.
Interesting about the EUFI bios settings, I really haven't played with any of them except the autotune, which worked amazingly.

256GB RAM ! :D - which brand? I was struggling to find 32gb dimms, I dare ask how much. When I looked I could have bought a wife, house and car for the same money.

Good to see it is being used to it's full potential, those excel spreadsheets and multiple tabbed browsers can bring the best supercomputer to it's knees :p
 
Associate
OP
Joined
8 Oct 2008
Posts
1,272
Eek your not having much luck Colin.

I have no doubt that your ram is the issue you need ECC for SMP, and also you'l need a pair at least for 2 cpu's. 1 stick probably won't work either. (I haven't tested that however from all past experience). As stated above - I'd be very suprsed if regular ram would work. The QVL is a bit sucky, as the crucial ram I have isnt listed, however the chips used (Crucial use micron chips) are listed. Anyways ECC registerd is what you need.

What colour are the LED's next to the CPU's ?
I too couldn't boot up initially which had me panicd. I don't recall the error codes though.
I did need a bios update to resolve that, which also had me head scratching for a bit. Only 1 special usb slot works for this process and ensure the filename is correct when saved to the usb stick. If I recall it did take a lot longer than I expected to flash, I almost gave up and rebooted mid way thinking it wasn't working (DON'T). Use a low capacity drive with FAT32
 
Associate
OP
Joined
8 Oct 2008
Posts
1,272
From a cursory glance that looks to be suitable. On a train atm so can't look up all the details.

I see the existing chips you have are listed as server ram.. a bit misleading imo as any "propper" server wouldn't use that. I guess a server can be any system, it's just the way it's configured.

http://www.crucial.com/csrusa/en/32gb-kit-%288gbx4%29-ddr4-pc4-17000-registered-ecc-12v-1024meg-x-72-p-ct4k8g4rfd8213

http://www.crucial.com/ProductDisplay?urlRequestType=Base&catalogId=10156&categoryId=&productId=155501&errorViewName=ProductDisplayErrorView&langId=-1&top_category=&parent_category_rn=&storeId=10151


Crucial stated to me on live chat both the above should work. Best check with them yourself too.

2 Dimms will give single channel mode
4 Dimms will give dual channel mode
8 Dimms will give quad channel mode


Also Colin - a real cold boot is often required if there is a previous boot failure (Turn off power at mains, turn on power at computer).
 
Last edited:
Associate
OP
Joined
8 Oct 2008
Posts
1,272
Appears to be 1.03 It was the first available board in Europe I could get. - and you ?
Have you heard about any differences with revisions ?

I am running bios version Version 0602, there is a 0702 which I have not installed on the basis of "If it's not broken - Don't fix it" The change log's for the bios data makes little sense.

Apparently however It adds
- CPU and memory voltage control is in
- Memory frequency control is in

They also added tons of additional options for performance tweaking.
 
Last edited:
Associate
OP
Joined
8 Oct 2008
Posts
1,272
:D Yeay

Good news Colin!
I was feeling pretty bad for you with so much awesome kit not working, glad you have it up and running.

I did update the bios to 0702 with no problems.
There are more options available, however I have not had a play with them. As per useual, they have made up specialist terms for various settings with little/no explanation to what they actually do :p. Plus I'm no bios expert so it likely me being uneducated in such things too. - Share if you find some special settings.

I did upgrade the ram to 128GB and chucked in a few 980's also, can't say I notice any difference in speed with quad channel not that I have bench-marked anything, at least I can do a few more things without running out of memory though.
The great thing is that the board is rock sold stable and fast!, its been running over 4 days at 100% CPU for the latest calc, never had one blip.

If anyone finds a app that allows you to monitor temps etc that would be handy. CPU'z stuff doesn't appear to report correctly. so I used the hand test, things are just re-reassuringly mildly warm which I assume is good.

And the tweak town review.. :o A bit OTT with the scores, however It's defiantly the best motherboard I have ever had :D. Just wish the IKVM was included, as I can't find where to get it.
 
Back
Top Bottom