Network backbone upgrade - critique welcome

[Rainmaker]

My current network is nice enough and functional, but ugly from a redundancy and design point of view. I currently have a flat network:

ISP: YouFibre business 2000 symmetric FTTP with /29 IPv4 routed over /31, and /48 IPv6.

Router: Whitebox x86 - Beelink EQ12 with 2x 2.5Gb Intel I225V (B3 revision, aka fixed 'I226' edition), Intel N100 4 core CPU and 16GB RAM.
Running OPNsense (soon VyOS, now my config script is written) with fq_codel QoS - hence the slight drop in headline speed to control latency under load.

Access switch - downstairs: YuanLey 9 port PoE(at) switch with 8x 2.5Gb copper and 1x 10Gb SFP+

} Uplinked over 20 metres of single mode LC fibre with 5dB attenuator, BiDi QSFPTECH 10Gb transcievers (Note to Dons: That's the OEM not a competitor)​

Access switch - upstairs: YuanLey 10 port switch with 8x 2.5Gb copper and 2x 10Gb SFP+

WiFi: Ruckus R710 Dual-band 4X4:4 802.11AC Wave2 AP, powered over PoE(at) from the downstairs switch

I have most of the homelab stuff upstairs. Proxmox server (AMD Ryzen 5500U 6c12t 32GB RAM - running Podman for AGH-sync and Vaultwarden, *arr stack, SABnzbd, Emby/Jellyfin/Plex), Synology NAS (40TB for media storage over NFS), Radxa ROCK 5B running Armbian for secondary DNS, Raspberry Pi 3B+ running AlmaLinux 9.6 and currently unemployed.

The new network is only in the planning stages, but my current proposal is to move to a proper collapsed core (tier 2) network with a solid 10Gb core/distribution switch (Mikrotik CRS305-1G-4S+IN maybe?). I can then move the two existing 2.5Gb units to being straight access switches. That way I can move the existing AOC single mode fibre uplink to the new core switch, add in a multimode or DAC link for the other switch and add one for the router. The router itself will be upgraded to either an Alta Labs Route10 (dual SFP+, 10Gb firewall and NAT with 10Gb IPS) or a CR3000 whitebox with Intel X710-DA2 or Mellanox ConnectX-4 to give scope for WAN upgrades to 10Gb and beyond, at which time I only have to upgrade the two access switches as LAN devices/servers catch up on the NIC side.

Core/distribution switch wise, the CRS305-1G-4S+IN seems the best fit, with a small form factor and multiple SFP+ ports at only £85. Open to ideas, however?

WiFi by the venerable Ruckus Unleashed enterprise box is currently 'fine' (>500Mbps per client) but will be upgraded to WiFi 7 as time and budget permits.

I'm running DNS on-prem with AdGuard Home (DoT to upstreams, DoH, DoQ and DoT to clients via iOS profiles etc). Locally, split DNS pushes traffic to the LAN IP of the DNS server, and globally dns.mydomain.com has A and AAAA records for the 10Gb VPS in Manchester, which has <2ms ping to my home and runs Nginx to load balance DNS traffic. If my home DNS goes down, it falls back to my secondary home DNS. If that (or WAN!) fails, the VPS itself has an instance running to pick up the slack until my home servers are back up.

The proposed changes are roughly as follows:

Does anyone care to offer any critique, comment or ideas? I'd be glad of the feedback. Thanks!

 

[Caged]

There's not really a lot to add, it's a home network after all and a line needs to be drawn somewhere. I'd probably want to consider the Proxmox host, the lab guests running in it, and the *arr stuff as separate zones to prevent the lab stuff and the ISO downloads from being able to talk to each other, maybe handle this on-box rather than dragging it across VLANs down to a firewall, I don't know if you can do iptables stuff in Proxmox or if you'd deploy a vitual router on it, I guess it depends on your needs.

For resilience maybe add a 5G router that can speak VRRP to sit alongside the firewall box so that if the box dies you fail over to 5G, you could presumably have Wireguard VPN running on the main firewall and the 5G and have a routing protocol in the VPS environment fail over to the tunnel coming out of the 5G link if you lost your fibre. This should give you pretty much no impact of your ONT or main firewall blowing up, or someone digging it up.

 

[Rainmaker]

There's not really a lot to add, it's a home network after all and a line needs to be drawn somewhere.

Does it?

I'd probably want to consider the Proxmox host, the lab guests running in it, and the *arr stuff as separate zones to prevent the lab stuff and the ISO downloads from being able to talk to each other, maybe handle this on-box rather than dragging it across VLANs down to a firewall, I don't know if you can do iptables stuff in Proxmox or if you'd deploy a vitual router on it, I guess it depends on your needs.

Yeah, already done with nftables. The /29 is allocated into 2x DNS, 1x SNAT for DHCP LAN clients, and the rest are allocated to the servers (all with global IPv6 via a /64 cut from the /48) so they don't interact with the LAN clients.

For resilience maybe add a 5G router that can speak VRRP to sit alongside the firewall box so that if the box dies you fail over to 5G, you could presumably have Wireguard VPN running on the main firewall and the 5G and have a routing protocol in the VPS environment fail over to the tunnel coming out of the 5G link if you lost your fibre. This should give you pretty much no impact of your ONT or main firewall blowing up, or someone digging it up.

Now there's a shout. I hadn't considered that, and it's very doable. I already run WireGuard so I can get back into the LAN, but it hadn't occurred to me to run it behind the VPS so I get full failover. Thanks! Any experience of the Mikrotik stuff? Wondering if I should stick to offloaded enterprise gear or not, but Patrick at STH rates it and at the price...

e: a word

 

[Caged]

I like the Teltonika boxes for 5G duties

 

[Rainmaker]

Thanks again, the TRB500 looks like the ticket.

 

[ChrisD.]

Personally, I would replace your router and AP with ASUS devices. Have you seen their marketing material?

 

[Rainmaker]

Personally, I would replace your router and AP with ASUS devices. Have you seen their marketing material?

I knew this thread would deliver. Finally, some solid advice! Loads of antennae and blinky LEDs though, yeah? I like this one, it already comes with holes so you don't need to piddle around with DNAT.

 

[Duke]

Personally, I would replace your router and AP with ASUS devices. Have you seen their marketing material?

And replace the switches with one of these bad boys - "Tenda 8-Port 10/100Mbps Fast Network Switch" or better still a hub? The more collisions the better I hear

Nice network btw

 

[Rainmaker]

And replace the switches with one of these bad boys - "Tenda 8-Port 10/100Mbps Fast Network Switch" or better still a hub? The more collisions the better I hear

Someone told me if you get a hub and give every device the same IP address, it makes your network wicked fast because you don't ever need to look up the ARP table, you already know the address you need. Then, when Asus' router gives your network malware, it gets confused because everyone has the same address, so it's a win/win! I think @Avalon tried it once, and now his network is so fast it fell off the ISP, and that's why he hasn't replied yet.

Nice network btw

Thanks. I realised last night I have an Openreach ONT, so instead of going 5G for failover I can bag the cheapest 10Mbps or whatever bargain basement package and just use that as failover. It puts single point of failure at the router, but as Caged said we do have to finally draw a line somewhere!

 

[Avalon]

Someone told me if you get a hub and give every device the same IP address, it makes your network wicked fast because you don't ever need to look up the ARP table, you already know the address you need. Then, when Asus' router gives your network malware, it gets confused because everyone has the same address, so it's a win/win! I think @Avalon tried it once, and now his network is so fast it fell off the ISP, and that's why he hasn't replied yet.

You joke, but I recently posted to my ISP's support forum asking as the founder had left with zero notice following a suspicious migration and that at exactly the same time everyone had to set-up new DD's as if we had switched back end billing systems, and as exactly as much progress had been made on enabling IPv6 as reinstating the ability for new customers to order service for the last 6 months, did they have a future plan, or were we just sold off on the sly? So it's not out of the question for me to drop off the network. Only took them over a fortnight to delete my post without replying, so I don't know what I was worried about

I simplified my set-up and built in resilience a while back when moving everything back on prem. It doesn't actually look that different to yours in principal (i'll run out of IDS at 3.5Gb/s, NAT about 8Gb/s, Wifi6 vs 7, and I don't run anything off prem. now). First question is does everything have a UPS? Next up, I wasn't a fan of only using virtualized routing and for years clung to physical with a VM failover, then I moved to Proxmox which made things like WAN VLAN tagging (eg City Fiber) dead easy as you're just presenting it clean to the router distro, if you like to swap router distros regularly, or play before deploying, Proxmox is a great way to test/learn without becoming persona non grata to your family, in your own home. I found Mikrotik in general an up hill struggle with a few quirks, but it could be really flexible, the management options got changed significantly a few years back for the better. I'm only using an Aruba 1930 48p (without the cloud management) for my core because it was stupidly cheap and 4xSFP+ and 370w of PoE+ make life easy.

Failover is one of those areas that needs careful thought, some routers (and router distro's) will support for example a USB modem/dongle, but support can be very hit and miss depending on the specific modem chipset revisions etc. Making use of a pov spec connectivity option via ONT with OR is going to be a lot easier. I got as far as a ZTE-MF286D which is only 4G, but it's not ideal and is currently sat being a 4G router due to a cancelled fiber build elsewhere, so i'm running nothing at present.

In terms of where to stop? I don't think UPS/failover WAN is unreasonable at home if connectivity is important to you (ours is), I also have no wish to have my families DNS or browsing activities logged, not because I have anything to hide, but because I have both directly and indirectly seen how that can be, and is abused, and i'm really not a fan.

 

[Rainmaker]

Still having fun and games with the ISP then? You're with Yayzi iirc? I actually don't have everything on UPS yet, which is a glaring omission. I have been looking at Eaton 3S 850B or similar, it's enough run time to safely power everything down or keep things running for short blips. I'm not averse to virtualised routing these days (tech has come a long way), but I've never had the hardware to make it worthwhile. When I needed a new router I just grabbed an EQ12 cheap and installed OPNsense on it. If I'd gone for a box with 3 or more interfaces I'd have been just as happy to use one for management and pass through two to Proxmox or similar. If I go x86 next time that's likely a route I'll take, though virtualised 10G is probably something that will take more reading. The ability to snapshot, roll back and have automated backup and restore is a very nice thing. What are you using for routing atm? Did the UDMP-SE not materialise or get boring? I know, like me, you've gone through the gamut and back again. Sometimes a change is as good as a holiday and all that.

When it comes to privacy and controlling your own data, I'm right there on the bus with you. My ISP should be a dumb pipe, and when government decides they should be logging DNS queries and sites visited in case Doris at the local library fancies a warrantless look at what we like to do online, I'm oot. Twenty years ago it would have seemed ridiculous to care, in a sense. These days you're more likely to get your door kicked in for something you saw or said on social media than for mugging an actual granny, so...

 

[Caged]

I'd skip a home UPS for network gear at this point unless we're talking about spending a really small amount on it. If you have the budget and space available then look at home battery systems. Use it to reduce your peak electricity spending and at the same time give you some circuits around the house that are backed up - kitchen outlets for your fridge/freezer and to plug in a laptop to charge, some lighting, your comms gear.

With a lot of these battery systems it's a simple addon to put a string of solar panels on, the things are £60 each now for a 400w panel which is mad.

 

[Rainmaker]

I'd skip a home UPS for network gear at this point unless we're talking about spending a really small amount on it. If you have the budget and space available then look at home battery systems. Use it to reduce your peak electricity spending and at the same time give you some circuits around the house that are backed up - kitchen outlets for your fridge/freezer and to plug in a laptop to charge, some lighting, your comms gear.

With a lot of these battery systems it's a simple addon to put a string of solar panels on, the things are £60 each now for a 400w panel which is mad.

I was thinking along the lines of £90 on a small Eaton system for the NAS, Proxmox box and router etc. If nothing else it gives a few minutes' grace to stop the raid array hard powering off in the event of a power cut. On that note (sort of), the old Synology is nearing EoL in the literal sense. I have 40TB of media and photos etc backed up (with copies off-prem) so I'm wondering whether to 'just' move the drives to a caddy attached to Proxmox over USB-C and pass it through to a guest and rebuild it that way. Currently the NAS is basically just serving storage to Proxmox guests over NFS anyway so a direct attach caddy would cut out the middle man.

 

[Avalon]

Still having fun and games with the ISP then? You're with Yayzi iirc? I actually don't have everything on UPS yet, which is a glaring omission. I have been looking at Eaton 3S 850B or similar, it's enough run time to safely power everything down or keep things running for short blips. I'm not averse to virtualised routing these days (tech has come a long way), but I've never had the hardware to make it worthwhile. When I needed a new router I just grabbed an EQ12 cheap and installed OPNsense on it. If I'd gone for a box with 3 or more interfaces I'd have been just as happy to use one for management and pass through two to Proxmox or similar. If I go x86 next time that's likely a route I'll take, though virtualised 10G is probably something that will take more reading. The ability to snapshot, roll back and have automated backup and restore is a very nice thing. What are you using for routing atm? Did the UDMP-SE not materialise or get boring? I know, like me, you've gone through the gamut and back again. Sometimes a change is as good as a holiday and all that.

When it comes to privacy and controlling your own data, I'm right there on the bus with you. My ISP should be a dumb pipe, and when government decides they should be logging DNS queries and sites visited in case Doris at the local library fancies a warrantless look at what we like to do online, I'm oot. Twenty years ago it would have seemed ridiculous to care, in a sense. These days you're more likely to get your door kicked in for something you saw or said on social media than for mugging an actual granny, so...

Short version as it's probably better for PM: UDM-SE is OK, it's limitations are becoming a slight concern - 3.5Gb/s for filtering less so than 1Gb/s for PPPoE, but i've read claims that may have improved. NVR and - stupidly expensive - doorbell are actually better than I had hoped, objectively it's not any better than anything else, but it's probably saved me a fair chunk of time setting up and stringing together various functions.

UPS wise I tend to buy used units without batteries and then order in a new cell pack. My current UPS' are older APC 1500 or above units, the Eaton stuff is generally more linux friendly afaik, but my servers are set to start shutdown after 5 mins on UPS, the basic network will run for hours after that. Caged may have a point in terms of wider usage/convenience of a portable battery unit, I know someone on a canal boat who loves them, perhaps less so his employer who doesn't realise he's charging one of them 5 days a week My only concern would be the battery packs in a few years, are they replicable, what's the cost/availability like or is it a new unit? My APC is both easy/inexpensive to get batteries for from a number of vendors, and ties in nicely to my ecosystem.

Based on your other reply, i'd be considering if ZFS under Proxmox would be the best solution to your storage issue. I like because it makes VLAN tagging (eg CF WAN) trivial, often easier than trying to mess about with each distro to sort it and get online, but it's also great for test environments and more importantly, it also handles ZFS quite nicely, and if you're only dealing with 40TB, it's quite easy to incorporate that into what you already run. Of course TNS or UR can be virtualised if you want to go that route, as can DSM technically... i've just never been a Synology fan. Also not a fan of USB based storage, but that's just me.

 

[Semple]

Personally, I would replace your router and AP with ASUS devices. Have you seen their marketing material?

I'm sure there's a guy on here who swears by them... He certainly makes a monthly thread asking why all the hate on Asus.

 

[Rainmaker]

I'm sure there's a guy on here who swears by them... He certainly makes a monthly thread asking why all the hate on Asus.

Shh, don't say his name into the monitor. He'll appear to tell us how nerdy, overly pedantic and wrong we are.

 

[Scott C]

Does anyone care to offer any critique, comment or ideas? I'd be glad of the feedback. Thanks!

I also assume that your diamond shoes are too tight and your wallet is too small for your fifties?

Seriously though, that looks a pretty solid setup As over-engineered and over-kill as all good home networks should be

Curious why you're moving away from OPNsense though?

 

[Rainmaker]

I also assume that your diamond shoes are too tight and your wallet is too small for your fifties?

Ha! You'd be amazed how cheap it all was, I just built it out over time. Switches were £50 each (Mikrotik 4x SFP+ 10G swiitch is £110), the router was £180 on offer, pre-made fibre patch cables are about £2-3 each, regular modules £5-10, and I got two of my servers on a price glitch from Amazon for £3.50 each(!). About the most expensive/convoluted aspect was the OS2 fibre backbone between downstairs and upstairs - £3.50 for 15M cable, £10 for conduit, £8 for two 'lead in kits' and £4 for external cable entry covers to keep the walls neat in and out, and £35 for two BiDi modules - so I asked for those for Father's Day lol...

Seriously though, that looks a pretty solid setup As over-engineered and over-kill as all good home networks should be


Curious why you're moving away from OPNsense though?

Honestly? And I know it's the wrong answer... but boredom/novelty. If you want one thing to be boring, it's your router. It should just work and keep on working year after year. And they do, which is boring. I've been using *nix for over 20 years at this point, so my backup server (mirrored twice offsite) has a nice folder under Software > Backups > Routers with config files for everything from OPNsense to OpenWRT, Sophos to VyOS, and plain old OpenBSD/FreeBSD (dhcpcd/dhcpd, pf.conf etc) and Debian.

When I see a changelog that looks tasty or I just get bored, I hose the box, scp over the config and I'm back up and running for a few months/years until I get the itch again. Easy. I've been particularly impressed with OpenWRT in the past, but OPNsense is really very nice to work with. It's mostly rock solid, has great throughput and nice plugins and monitoring. It was a doddle to set up my /29 over /31 and the /48>/64 as well, which is nice. VyOS is something I've used before and I really like it.

I'm more curious as to what differences I might find in overhead, resource usage and throughput/latency at 2 and 10 Gb versus a *BSD box, so change it is. I have a TIG stack running in an Alpine LXC under Proxmox listening to the router address. Whatever OS/distro I put on there, I configure SNMP and keep on collecting the bandwidth and data stats, so it's pretty vendor agnostic.

 

[Caged]

If you want to play around then add another switch and do MC-LAG, though there are the usual nuances that come with it being Mikrotik kit.

 

[Rainmaker]

If you want to play around then add another switch and do MC-LAG, though there are the usual nuances that come with it being Mikrotik kit.

More kit. My wife thanks you! Great idea though.