Hi all,
After gaining my VCP6.5-DCV I decided to pursue the VCP6-NV. To do so, I decided to not bother with a nested environment and go for a proper homelab capable of 10Gbit. Using the VMUG subscription gives me access to vSphere Enterprise Plus, vSAN, NSX, vRo licences etc so I thought I'd build a little vSAN cluster.
I know a lot of people run Intel NUCs, but they lack 10Gbps NICs and are limited to 32GB or RAM per node. A vSAN enabled host consumes around 10GB of RAM initially, plus the 10GB requirement for the vCenter server doesn't leave much headroom for other VMs. While great little boxes, I wanted something capable of more. The hosts I am using are SuperMicro SYS E200-8D micro servers. They are small (think thin client size), have a 6 core/12 thread 1.9 Ghz Xeon processor, take up to 128GB of ECC RAM (or 64GB non ECC), feature two 10Gb NICs as well as another two 1Gb NICs, plus IPMI which allows remote console access. They're not cheap sadly, coming in at around £800 a node for which doesn't come with RAM or any HDDs.
Luckily I managed to get hold of some cheap RAM which gave me 64GB in total for each host. I then bought three Crucial 1 TB SSDs to act as the vSAN capacity tier and three 250 GB Samsung nVME SSDs for the caching tier.
Finally I needed a switch capable of 10 Gbps. Since I was already invested in the UniFi range, I went for the 16-XG. This gives me 3x 10Gbit RJ45 interfaces since the servers are copper only plus a 4th as a downlink to the rest of my network. Already having bought a QNAP NAS, I bought a dual 10Gb SFP+ PCI card and some DACs which occupy 2 of the remaining SFP+ ports. The NAS acts as a Plex server as well as some automation for home media. The reason I wanted it at 10Gb is to quickly backup the VMs using Veeam, plus also use it as a datastore if I've managed to break the vSAN storage. Throughput wise on the NAS I see between 250-350 MB/s which isn't too shabby for four spinning rust 6TB WD Reds.
Unfortunately the SuperMicro servers are far from quiet, I would not recommend them if you're having them in the same room unless you like 40mm fan noise. You can fit Noctua fans, but since the CPUs heatsink is passive, they don't provide enough static air pressure to keep the CPU cool under medium and higher load, so stick with the stock ones! SuperMicro do offer a HSF assembly which does fit, however I haven't tried it yet.
I did eventually pass my VCP6-NV, next on my list is VCAP deploy in both the DCV and NV tracks, although I'm going to wait until March for the new 2019 exams to come out.
This is how it all looks at the moment after moving it all upstairs to a spare bedroom.
Once the door is closed I can't hear any fan noise so I'm happy it's moved up there from being in the corner of the living room in a tiny closet getting somewhat toasty!
This is what the vCenter environment looks like. It's pretty baron at the moment as I've been building NSX while blogging it which adds a fair amount of time to the whole process! As you can see I'm already consuming ~82 GB of RAM across the cluster which does not leave much room if I were limited to 32GB of RAM per host. However to be completely fair it is possible to reduce the RAM that vCenter requires as well as remove the NSX VM memory limits.
I have started a blog if anyone wants to have a look, it's early days yet and the articles I've written so far need a little work still but have generally received positive feedback on Reddit. When vExpert 2020 opens I hope to apply and be awarded it, which would be extremely cool.
https://virtualinsanity.org/
The next step is to get a small 19" cabinet and tidy it all, maybe something on casters so it can be wheeled around as I may at some point do some demonstrations etc with it.
If anyone has any VMware questions please let me know, I've been working with it for over 10 years now so I've learnt a fair amount!
Thanks for reading.
After gaining my VCP6.5-DCV I decided to pursue the VCP6-NV. To do so, I decided to not bother with a nested environment and go for a proper homelab capable of 10Gbit. Using the VMUG subscription gives me access to vSphere Enterprise Plus, vSAN, NSX, vRo licences etc so I thought I'd build a little vSAN cluster.
I know a lot of people run Intel NUCs, but they lack 10Gbps NICs and are limited to 32GB or RAM per node. A vSAN enabled host consumes around 10GB of RAM initially, plus the 10GB requirement for the vCenter server doesn't leave much headroom for other VMs. While great little boxes, I wanted something capable of more. The hosts I am using are SuperMicro SYS E200-8D micro servers. They are small (think thin client size), have a 6 core/12 thread 1.9 Ghz Xeon processor, take up to 128GB of ECC RAM (or 64GB non ECC), feature two 10Gb NICs as well as another two 1Gb NICs, plus IPMI which allows remote console access. They're not cheap sadly, coming in at around £800 a node for which doesn't come with RAM or any HDDs.
Luckily I managed to get hold of some cheap RAM which gave me 64GB in total for each host. I then bought three Crucial 1 TB SSDs to act as the vSAN capacity tier and three 250 GB Samsung nVME SSDs for the caching tier.
Finally I needed a switch capable of 10 Gbps. Since I was already invested in the UniFi range, I went for the 16-XG. This gives me 3x 10Gbit RJ45 interfaces since the servers are copper only plus a 4th as a downlink to the rest of my network. Already having bought a QNAP NAS, I bought a dual 10Gb SFP+ PCI card and some DACs which occupy 2 of the remaining SFP+ ports. The NAS acts as a Plex server as well as some automation for home media. The reason I wanted it at 10Gb is to quickly backup the VMs using Veeam, plus also use it as a datastore if I've managed to break the vSAN storage. Throughput wise on the NAS I see between 250-350 MB/s which isn't too shabby for four spinning rust 6TB WD Reds.
Unfortunately the SuperMicro servers are far from quiet, I would not recommend them if you're having them in the same room unless you like 40mm fan noise. You can fit Noctua fans, but since the CPUs heatsink is passive, they don't provide enough static air pressure to keep the CPU cool under medium and higher load, so stick with the stock ones! SuperMicro do offer a HSF assembly which does fit, however I haven't tried it yet.
I did eventually pass my VCP6-NV, next on my list is VCAP deploy in both the DCV and NV tracks, although I'm going to wait until March for the new 2019 exams to come out.
This is how it all looks at the moment after moving it all upstairs to a spare bedroom.
Once the door is closed I can't hear any fan noise so I'm happy it's moved up there from being in the corner of the living room in a tiny closet getting somewhat toasty!
This is what the vCenter environment looks like. It's pretty baron at the moment as I've been building NSX while blogging it which adds a fair amount of time to the whole process! As you can see I'm already consuming ~82 GB of RAM across the cluster which does not leave much room if I were limited to 32GB of RAM per host. However to be completely fair it is possible to reduce the RAM that vCenter requires as well as remove the NSX VM memory limits.
I have started a blog if anyone wants to have a look, it's early days yet and the articles I've written so far need a little work still but have generally received positive feedback on Reddit. When vExpert 2020 opens I hope to apply and be awarded it, which would be extremely cool.
https://virtualinsanity.org/
The next step is to get a small 19" cabinet and tidy it all, maybe something on casters so it can be wheeled around as I may at some point do some demonstrations etc with it.
If anyone has any VMware questions please let me know, I've been working with it for over 10 years now so I've learnt a fair amount!
Thanks for reading.
Last edited:
