Advice on homelab/homeprod and containers/setup

TechMinerUK

TechMinerUK

TechMinerUK

Soldato
Joined
29 Dec 2014
Posts
2,587
Location
The "North"
Hey all,

Recently I've been thinking of how I can move my homelab/homeprod setup forward as whilst I revamped it a little while ago and it's not really as "good" as what I wanted it to be.

To cut a long story short, I'm looking to move more stuff back "on-prem" at home where possible and not silly, so local file storage (NextCloud), local picture storage (Immich) and local media (Emby) along with my home automation stuff.

Right now I have the following hardware
  • 3x Minisforum MS-01 (64GB RAM, 12600H, 1TB local storage) - Used as a Hyper-V cluster and DR host
  • 2x QNAP TS932PX (1x with 4x 12TB disks for media/file storage and 4x 2TB SSDs for VM storage and the other with 2x 24TB HDDs, 2x 6TB HDDs and 4x 1.8TB HDDs for VM storage)
  • 1x Beelink (N100, 16GB RAM, 512GB SSD) - Emby physical server
  • 1x GMKTec (N100, 16GB RAM, 256GB SSD) - Zigbee2MQTT
Currently in terms of VMs I have:
  • 3x Windows Server DCs (1 on each host)
  • 1x Windows Server CA
  • 1x Windows Server (Veeam Backup & Replication)
  • 1x Windows Server (File server)
  • 10x Ubuntu (NextCloud, Sonarr/Radarr, UptimeKuma, Postfix/Email notifications, VPN, Ansible [Not yet used], UniFi controller, Public Reverse proxy, Internal reverse proxy, Docker [Not yet used])
  • 1x Home Assistant OS

Overall I feel as though the above is overkill which is fine but the idea initially was a lower power setup that didn't lack "grunt" when it came to spinning up a VM but also wasn't excessive to the point of being a pain to manage etc however whilst the MS-01s are great they are hot and feel overkill and not really super for the task at hand which is continuous operation whilst sipping power. Likewise the above setup has various SPOFs such as the 10G switches and NAS'.

I do want to keep the hosts running Hyper-V, not because I don't want to learn but because I can't find a suitable hypervisor that supports FDE in a polished format (I know Proxmox can "work" with a custom Debian build + a Clevis/Tang network unlock setup or using a light SSH to input it on reboot but that’s not ideal for this setup).

I've read that Docker/Podman could potentially be a way to move to smaller hosts e.g. having multiple Docker containers on a VM rather than a VM per role however I've ended up with my head swimming after all the reading on how to securely configure containers with networking and rootless operation (Which seems to break many containers by default)

Ultimately I was potentially thinking of something like so:
  • 2-3x N300 Mini PCs (32 or 64GB RAM each with larger SSDs)
  • 2x NAS (Similar config as now but one directly hosting SMB rather than iSCSI for storage) or some sort of custom build server using a Jonsbo case
  • A 2.5G switch (rather than the 10G ones) for connection between the hosts and storage
In terms of the way forward, for the Windows VMs there isn't really much I can do to streamline these other than bop the third DC on the head and remove the need for a file server by moving that to its own dedicated box (Direct on the NAS or a custom TrueNAS setup) but for the NextCloud, Immich, etc I think I may have potential room to consolidate these in a singular VM using Docker but does anyone have any suggestions on how to securely achieve this (I will profess that I am a complete amateur when it comes to Docker and I'm very paranoid when it comes to security)

Any help, advice, suggestions would be appreciated :)
 
I would be surprised if a 16 core 32 thread Ryzen with 128GB of RAM couldnt run nearly all of that in a server tbh.

I would go for 1 NAS with a 10GB NIC to the server (no switch). But make the NAS slightly oversized so you could at least run a couple of VMs on it if needed. DIY NAS isn't too challenging these days, just get a case with lots of 5.25 bays so you can add hot swap caddies so disk failures are easy to deal with.

If you need to backup the NAS as well then I would look at streaming it off to an S3 bucket (very cheap if you just writing it there and only pulling back if you have a catastrophic failure with the NAS)

Any other networking just do over 1GB with a cheap low power switch?
 
BongoHunter said:
I would be surprised if a 16 core 32 thread Ryzen with 128GB of RAM couldnt run nearly all of that in a server tbh.

I would go for 1 NAS with a 10GB NIC to the server (no switch). But make the NAS slightly oversized so you could at least run a couple of VMs on it if needed. DIY NAS isn't too challenging these days, just get a case with lots of 5.25 bays so you can add hot swap caddies so disk failures are easy to deal with.

If you need to backup the NAS as well then I would look at streaming it off to an S3 bucket (very cheap if you just writing it there and only pulling back if you have a catastrophic failure with the NAS)

Any other networking just do over 1GB with a cheap low power switch?
Click to expand...
Any particular Ryzen CPU worth looking at for this sort of workload (Not had much experience outside of desktop gaming) :)

NAS direct to the PC isn't a bad shout as it would cut down on a lot of the switches since HA wouldn't be needed anymore, I'm really tempted by the Jonsbo stuff atm as it looks decent when paired with the right hardware :)
 
TechMinerUK said:
Any particular Ryzen CPU worth looking at for this sort of workload (Not had much experience outside of desktop gaming) :)

NAS direct to the PC isn't a bad shout as it would cut down on a lot of the switches since HA wouldn't be needed anymore, I'm really tempted by the Jonsbo stuff atm as it looks decent when paired with the right hardware :)
Click to expand...
Assuming you don't need really high clock speed or ECC then the Ryzen 9 5950X is a bargain at £260
 
You have plenty of kit there to do what you want. I would recommend maybe setting up one of the machines in isolation and playing about with docker or podman until you’re more comfortable. Watch some videos on YouTube, there are lots on these topics.

If I understand right, you’re using a whole VM for each service which isn’t efficient or usually necessary. I wouldn’t spend any more money until you do a bit more learning/experimentation. I can’t see what other purchases would really be necessary anyway for what you describe, assuming you containerise everything that suits it. You can then review performance and see if hardware needs any more power.
 
alex24 said:
You have plenty of kit there to do what you want. I would recommend maybe setting up one of the machines in isolation and playing about with docker or podman until you’re more comfortable. Watch some videos on YouTube, there are lots on these topics.

If I understand right, you’re using a whole VM for each service which isn’t efficient or usually necessary. I wouldn’t spend any more money until you do a bit more learning/experimentation. I can’t see what other purchases would really be necessary anyway for what you describe, assuming you containerise everything that suits it. You can then review performance and see if hardware needs any more power.
Click to expand...
I've currently got a VM in isolation that I just revert the checkpoint on regularly, Docker has definitely proven to be an absolute sinkhole for my time as there are so many ways to configure it and the securing of it seems to be an absolute minefield!
 
TechMinerUK said:
Is the 5950X not power hungry? I must admit I'm keen to try keep the power usage "fairly low" where I can :cry:
Click to expand...
It's 105w TDP

30 watts at idle - probably about 120w+ at full load, more if it using PBO

If you look at AMD Epyc you can get a 16 core with a 65w TDP, e.g the Epyc 4545P - but they cost more than the Ryzens, but you do get ECC support and other enterprisey stuff
 
TechMinerUK said:
I've currently got a VM in isolation that I just revert the checkpoint on regularly, Docker has definitely proven to be an absolute sinkhole for my time as there are so many ways to configure it and the securing of it seems to be an absolute minefield!
Click to expand...
Look at a container orchestrator like K3s or one of the K8s distributions - there's a lot of features they offer that you don't need with a single node (e.g clustering), but you gain a good way to manage your containers and access (rancher desktop might work for you if your running on a single host).
 
BongoHunter said:
Look at a container orchestrator like K3s or one of the K8s distributions - there's a lot of features they offer that you don't need with a single node (e.g clustering), but you gain a good way to manage your containers and access (rancher desktop might work for you if your running on a single host).
Click to expand...
I'd like to learn how to do the high availability configuration (Even if its just VMs) so it looks like I need to continue down the rabbit hole of containers, it just seems to be a minefield with lab stuff as much of the things I have tried so far (Immich & Nextcloud) the moment I tried to harden the VM just stopped working :D

Armageus said:
You can lock the 5950x to a 65w TDP with Eco mode anyway I believe.
Click to expand...
I'll have a look into that as one of the main reason I went with the MS-01 units was to reduce power, that being said whilst they are great they run a bit hot for my comfort and compared to the ML110 I had previously they have an annoying hum which I've started to noticed in the office when I'm working :D
 
You must log in or register to reply here.
Back
Top Bottom