Proxmox

thenewoc · 13 Aug 2020 at 10:46

Am tentatively planning a new build later this year when X670 and the next Ryzen CPU's release and thinking of putting together a home lab based on Proxmox.

Anyone done this and can you offer any advice with regards to using Proxmox as a ZFS NAS device and GPU pass through to a Windows VM as the main two core features for the build?

Originally I was thinking I would have to use something like FreeNAS inside a Proxmox VM but I think that was before realising that ZFS is in Proxmox anyway. Is it possible to just use Proxmox as the NAS? I don't want to use specifically a NAS OS as I want a home lab that can serve the purposes of being a NAS and creating some VM's and Docker containers. I didn't like the VM / jails side of things in FreeNAS so that's why I wasn't considering running FreeNAS on bare metal and I don't think it does GPU pass through.

Am also a bit confused over overall drive configurations such as using a pair of SSD's or M2 drives for the Proxmox installation, some form of Raid-z? Should this only be big enough for the Proxmox installation or can other stuff be stored on this too such as VM's? Can / should any SSD's be used to cache the spinning rust?

I don't need masses of storage, I was thinking a few 4TB drives but realise I'd loose some of these to protecting this array. What would be the sweet spot for ~8TB of usable space and could this be added to fairly easily if this requirement were to grow?

Thanks in advance for anyone's insights...

thenewoc · 13 Aug 2020 at 11:31

picnic said:
Sorry I can't help but I'll watch this topic with interest. I do run a Proxmox server running several VMs, piHole etc. but not using it as a NAS/Storage. As my hardware is also a 1u rackmount server I got 2nd hand I guess the GPU pass through is not something I can do anyway.

No worries. I looked at rack mount cases but as this is for a small home office I've decided to repurpose my existing case (Fractal Design Midi Tower) which has a fair amount of 3.5" drive trays. I had some ideas for rack mount 4u cases but as I have a sit stand desk I'd have to build quite a bespoke desk pedestal that could accommodate the leg of the desk.

Do you have your Proxmox installed on a pair of drives in Raid-z1 and do you have your VM's installed on these too or on separate drives?

thenewoc · 13 Aug 2020 at 11:56

Some further background...

Originally I was looking for a dual build midi case such as a Phanteks P600S but didn't like the fact that the second system was confined to a MITX size. It would mean building the server on a MATX as more choice available in that size which would leave the separate desktop build being confined to a MITX. Part of wanting the P600S was to have a windowed case that I could build the main system to look nice with some RGB but this would largely not be possible with the server build being in the vertical position on view, except I could mount the GPU vertically in front of part of it. I then also thought if this was going to house a number of disks that a windowed cases wouldn't be so sound proofed. I was then given the idea of rack mounting in a different thread in the cases forum and did quite a bit of research into rack mounted cases (4U). The Logic Case SC-415A seemed to fit the bill on value and has the option for 5 or 10 hot swap bays depending on whether you were going to include a GPU or not. However, if I had two of these I would need to custom build a desk pedestal out of plywood to match the desk and to get the size of the 4U cases under the desk would need to side mount them so each could extend back under the desk and either side of the desk leg. Providing I can run Windows in a Proxmox VM with GPU pass through it doesn't seem like I would any longer need two completely separate systems.

thenewoc · 14 Aug 2020 at 16:50

thanks for your insights @picnic @Buffalo2102

What I'm a bit confused about and can't find details for is does Proxmox benefit from being installed on an SSD or M2 drive and can that drive / should that drive also be used for storing VM's? If I use a couple of SSD's or M2's in ZFS Raid-z1, can this be partitioned to provide space for VM installs? Am just thinking of the current limitation on the number of M2 motherboard slots and whether I could buy two very large M2's but if the Proxmox install doesn't really benefit from the extra speed I could put that on an SSD ZFS Raid-z1. If you can mix the Proxmox and VM installs on the same M2's then I think this would be preferable as it would leave the most number of SATA ports free.

For the HDD's if these are configured in a ZFS pool would / can the performance be enhanced with an SSD cache of some type?

thenewoc · 31 Aug 2020 at 19:07

thanks @Buffalo2102 is that screenshot just showing pools? Bit confused what the local one is, is that just a completely separate VM you didn't want on local-lvm? I'd be using some NAS HDD for some form of storage raid.

thanks @Doctor McNinja, just at the preliminary stages of planning a new build but waiting on X670, Ryzen 3 / Vermeer CPUs and some 3200 MT/s Unbuffered ECC DDR4 ram which I think is coming later this year.

My thoughts so far based on the Asrock Rack X570D4U;

4 DDR4 ECC UDIMMS (up to 32GB per DIMM) - this needs clarification that motherboard/CPU support ECC error reporting

1 PCIe 4.0 x16 - GFX
1 PCIe 4.0 x1 - blocked by GFX card in x16 slot
1 PCIe 4.0 x8 - unused*

* so ideally if I was going to have an L2ARC I'd need a NVMe PCIe adapter here to get the next best speed after RAM with another M.2 drive, possibly also one that is PCIe 4.0 gen. I'm hoping this slot on an upcoming X670 variant becomes x16 with bifurcation support which would then support up to 4 M.2 drives here.

M.2 (PCIe4.0* x4 or SATA 6Gb/s); Form factor: 22110/2280/2260/2242 [CPU] - Pioneer M.2 (PCIe Gen 3 x 4) APS-SE20G [TLC] size to be decided
M.2 (PCIe4.0* x4 or SATA 6Gb/s); Form factor: 2280/2260/2242 [X570] - Pioneer M.2 (PCIe Gen 3 x 4) APS-SE20G [TLC] size to be decided

When I was looking at M.2 drives I came across an article (https://www.overclock.net/forum/355...d-value-ssds-2019-1tb-phison-e12-toshiba.html) that was recommending those that had a Phison E12 controller not sure if this is still the best, seemed also that some models can have different specs that may have substituted the controller for a different one too. The Pioneer's seem like a reasonable choice APS-SE20G [TLC]. Now I was thinking two of these mirrored but does the Proxmox install process allow you to establish a software RAID by selecting the two M.2 drives during the OS install or should the motherboard be doing this? These would be for Proxmox OS and VM's with backups being stored on the spinning rust VDEV1.

SATA3 #1 - 4TB IRONWOLF NAS 5900RPM - VDEV1 (RAIDz2)
SATA3 #2 - 4TB IRONWOLF NAS 5900RPM - VDEV1 (RAIDz2)
SATA3 #3 - 4TB IRONWOLF NAS 5900RPM - VDEV1 (RAIDz2)
SATA3 #4 - 4TB IRONWOLF NAS 5900RPM - VDEV1 (RAIDz2)
SATA3 #5
SATA3 #6
SATA3 #7
SATA3 #8

~8TB usable storage on VDEV1 with two drives providing parity protection?

ZIL & SLOG still need to do some research here. These seem like a good idea for added protection from power outage for instance but the later seems to be recommended over the former and presumably people don't have multiple Intel Optane drives for a SLOG so don't you then loose the added protection of something like a RAIDz1 on it?

Research sources:

RE: faster ECC memory, https://www.anandtech.com/show/15586/atps-ddr43200-industrial-dimms-up-to-128gb-12v-for-amd-intel

RE: primer on ZFS, https://www.servethehome.com/an-introduction-to-zfs-a-place-to-start/

RE: original mITX option: https://www.servethehome.com/asrock-rack-x570d4i-2t-amd-ryzen-server-in-mitx/ & https://forum.level1techs.com/t/asrock-x570d4i-2t/154306/73 & https://www.asrockrack.com/general/productdetail.asp?Model=X570D4I-2T#Specifications & https://www.anandtech.com/show/1545...570-motherboard-with-intels-10-gbe-controller

RE: original X470 mATX option: https://forum.level1techs.com/t/asrock-rack-x470d4u2-2t/147588/100 & https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U2-2T#Specifications & https://www.asrockrack.com/general/productdetail.asp?Model=X470D4U#Specifications & https://www.servethehome.com/asrock-rack-x470d4u-amd-ryzen-server-motherboards/

RE: current X570 mATX selection: https://forum.level1techs.com/t/asrock-x570d4u-x570d4u-2l2t-discussion-thread/159996 & https://www.asrockrack.com/general/productdetail.asp?Model=X570D4U-2L2T#Specifications & https://www.asrockrack.com/general/productdetail.asp?Model=X570D4U#Specifications