Setting up routing between subnets (with a twist)

[Rainmaker]

EDIT: I solved it. See post 2 for the solution if you're in a similar boat and find this via a search engine or the forum's search facility. Feel free to PM me with any questions or reply below.

I have just upgraded our x86 edge router to a re-purposed Dell Optiplex 7010 (4c8t Core i7 3770 @ 3.4GHz, 8GB DDR3 dual channel, 128GB SSD, Intel Pro 1000 VT quad port NIC). It's running Arch Linux with the linux-hardened kernel, using dnscrypt-proxy for DNS, dhcpd for DHCP, and Shorewall for NAT/firewall. I have the local network split so that my NAS and 4k CCTV cameras are on a separate interface and subnet (aka a 'DMZ' of sorts). The topology looks like this (except the second switch is now a Netgear ProSafe GS108):



The router itself has three of its ports physically connected (WAN, LAN and DMZ) and also connects to my VPN provider over WireGuard using virtual interface 'azirevpn-uk1'. If you're unfamiliar, WireGuard is a very small, fast and secure VPN protocol and behaves exactly like any other Linux interface. Don't let it being a VPN lead you down the mental garden path if you're unfamiliar. Treat it as any other interface (i.e. you use ip add, ip route etc as normal). The router is thus connected to my VM internet as well as the VPN over WireGuard. The LAN clients can therefore connect only to WireGuard via the router, and the DMZ is open to the internet via VM.

To tunnel the LAN via the WireGuard VPN I needed to manually set the routes after bringing up the interface (else the VM WAN/ISP link would be default). I did this by adding a table as so:

echo "10 vpn" >> /etc/iproute2/rt_tables

ip rule add unicast iif enp3s0f0 table vpn

ip route add default dev azirevpn-uk1 via 10.xx.xx.xx table vpn

Now the LAN clients all only get online via the VPN but the DMZ is free to serve over the net (so my domain still works when connecting to Plex etc). So far, so awesomesauce.

Unfortunately (for me) I have a dilemma. When the LAN is set up thus, all LAN traffic does indeed get tunneled through my VPN, but there's no route to the DMZ subnet so that LAN clients can watch Plex (for example). If I take down the WireGuard interface and delete the 'VPN table' route rules everything works again, and LAN clients can resolve and display content from the DMZ servers. This is of course because the router's default routing table allows subnets to talk to each other, but the VPN table I made doesn't - it only has one route, to the VPN tunnel.

I've learnt a LOT about networking in the last week (hence being able to build this router by hand myself and write all the conf files to get it working), but I'm still learning a lot - including routing. This is definitely a routing issue btw, as Shorewall is set up with policies to allow communication from LAN to DMZ (even though the reverse is disallowed for obvious reasons). Everything works if I remove the VPN and it's default routing entry (which again, doesn't include a rule for anything other than 'send all traffic to the VPN', so this is expected behaviour).

What's my best option here? I basically need to add a rule for LAN to say 'Yes tunnel everything, unless it's a request for 192.168.2.0/24 as that's local - ask the router at 192.168.1.1 instead'. Is it as simple as not setting the VPN table's dev azirevpn-uk1 route as default and then also adding a second route for 192.168.2.0/24 via 192.168.1.1 (the LAN's router IP)? If so which rule is default, azirevpn-uk1 or the one for 'all other traffic to the router for it to decide'? Or something else? Any help greatly appreciated. Thanks in advance!

 

[Rainmaker]

Never mind, I solved it on my own.

Instead of just adding the default rule to the VPN table, I added a second one like this:

ip route add default dev azirevpn-uk1 via 10.xx.xx.xx table vpn

ip route add 192.168.2.0/24 via 192.168.2.1 dev enp3s0f1 table vpn

This now gives a set of tables whereby the router itself can connect to the ISP, the VPN and both subnets (as is a router's job). All queries from the LAN clients skip the main routing table and hit the VPN table. This tells them that any traffic for 192.168.2.0/24 (the DMZ subnet) goes to it locally, but everything else goes via the VPN. When I posted the OP I was confused because I was trying to add the route as 'ip route add 192.168.2.0/24 via 192.168.1.1.... etc' because it was (to my mind) hitting the LAN router interface first. Once I twigged and switched it to the DMZ interface instead, Robert became my mother's brother.

BTW, I did also have to add some DNAT rules to Shorewall to allow local traffic between interfaces (which I'd already done, but I'm adding this in case someone finds this thread via search in future and wonders why their setup still isn't working!).

DNAT            loc     dmz:192.168.2.5         tcp     32400   -       $(WAN_IP_HERE)
DNAT            dmz     dmz:192.168.2.5         tcp     32400   -       $(WAN_IP_HERE)
DNAT            net     dmz:192.168.2.5:32400   tcp     32400

This tells Shorewall that (1) anything from the internet to port 32400 (Plex) is to be redirected to that port on the NAS at 192.168.2.5 in the DMZ zone; and (2) that any local traffic from loc (the LAN) or dmz (other DMZ clients) is also to be redirected to the NAS on that port. Failing to add the last two rules in (2) means people outside your network can connect in, but you can't resolve the services locally. The alternative is hairpin NAT, but this is nicer and avoids the traffic going through the firewall as external traffic. If LAN and DMZ weren't on separate physical interfaces, hairpin NAT and split DNS would have been needed.

If anyone wants my full config let me know and I'll post a blog or gist.
PS: Technically, @Avalon, me turning into such a huge network nerd and learning all this stuff is your fault for buying my lovely little APU2C4 and leaving me to run wild with bigger more complicated x86 boxes and hand built router OS'.... My wife would probably have something to say to you if she met you - I hadn't slept for a week sussing out all this. But now I know, and it's great.

 

[Rainmaker]

For completeness, I didn't want to type all this crap every time the system booted (ip route add... etc are temporary changes). For now the neatest way I know how to fix it is to make a shell script (wg-routing.sh) which is invoked at boot, after the network comes up, by a systemd unit/service I called wg-up.service

[Unit]
After=network.target

[Service]
ExecStart=/root/wg-up.sh

[Install]
WantedBy=default.target

#!/usr/bin/bash
ip link add dev azirevpn-uk1 type wireguard
ip address add dev azirevpn-uk1 10.xx.xx.xx/19
wg setconf azirevpn-uk1 /etc/wireguard/azirevpn-uk1.conf
ip link set up dev azirevpn-uk1
ip rule add unicast iif enp3s0f0 table vpn
ip route add default dev azirevpn-uk1 via 10.xx.xx.xx table vpn
ip route add 192.168.2.0/24 via 192.168.2.1 dev enp3s0f1 table vpn

In case I want to reset the WireGuard connection for any reason, I also made a small wg-down.sh script that can be run manually (before re-executing wg-up.sh or else issuing 'systemd restart wg-up' to get it back again).

#!/usr/bin/bash
ip link set down dev azirevpn-uk1
ip link del dev azirevpn-uk1
ip rule del unicast iif enp3s0f0 table vpn

Again, I hope this is of some help to someone, someday.

EDIT: The whole setup is now on Github.

 

[Steveocee]

Amazing read!

I'm surprised though that it didn't add it's own routes for directly connected networks. I won't profess to understand Linux networking at all but certainly on the 'Tik routers I use you get a default route for connected networks, I guess that isn't the case for Linux.

I've been reading about Wireguard as well, it's still under development right? Although very close to being included in the Kernel. It sounds a dream to set up for deployment and the performance figures are extremely encouraging.

Love that you've hand built and coded this, what was the drive behind it though? What made you think "Nah pfSense is too easy"?

 

[Avalon]

PS: Technically, @Avalon, me turning into such a huge network nerd and learning all this stuff is your fault for buying my lovely little APU2C4 and leaving me to run wild with bigger more complicated x86 boxes and hand built router OS'.... My wife would probably have something to say to you if she met you - I hadn't slept for a week sussing out all this. But now I know, and it's great.

If it makes you fee any better, the blame goes both ways... I was going to keep it simple, an APU with 1-2 FTTC connection(s) and some basic OpenVPN routing with an exception to route the local public services normally/via Cloudfare. Then instead of a second FTTC connection to load balance, BT kindly enabled g.fast. I say kindly, it’s turned into a needlessly complicated and drawn out ball ache that’s lasted weeks so far.

Obviously g.fast goes beyond the APU’s single OVPN tunnel speed limit. I’m now into this for a HeX, ER-X, the APU and have a sore i5 6400/Z270 and a bunch of 2T intel NIC’s ready to rock once BT finish screwing with my install. If I had any sense i’d have just virtualised pfsense and stuck to OpenVPN on fast cores, sadly reboots of a virtual environment for updates/upgrades aren’t my idea of fun on consumer (read non certified) hardware as downtime isn’t fun. WireGuard is amazingly quick, but i’m slightly concerned by the bleeding edge nature of the project, despite it’s base.

 

[Rainmaker]

Amazing read!

I'm surprised though that it didn't add it's own routes for directly connected networks. I won't profess to understand Linux networking at all but certainly on the 'Tik routers I use you get a default route for connected networks, I guess that isn't the case for Linux.

I've been reading about Wireguard as well, it's still under development right? Although very close to being included in the Kernel. It sounds a dream to set up for deployment and the performance figures are extremely encouraging.

Love that you've hand built and coded this, what was the drive behind it though? What made you think "Nah pfSense is too easy"?

Linux actually does add routes for directly connected networks, though IP forwarding is disabled by default (a simple setting change). For example 'out of the box' I could route between the subnets no problem at all. After adding the WireGuard connection/interface I could route to that also, from the router itself. For example, connecting to the router over SSH I could issue

curl ipinfo.io

and receive a reply that my IP was xx.xx.xx.xx Virgin Media. If I changed that to

curl ipinfo.io --interface azirevpn-uk1

I'd receive a reply that my IP was xx.xx.xx.xx 'AzireVPN network'. The 'problem' I had was that by default, using WireGuard's connection script 'wg-quick' would overwrite all routes on the box. This makes sense, as if you're a desktop user and connect to a VPN you definitely don't want your connection to leak! In my case, as a router, I actually do want it to 'leak' by design - only some stuff (aka my LAN) needs to be tunnelled. So by adding the tunnel manually (ip link add dev azirevpn-uk1 type wireguard... etc) I avoid the addition of automatic routing by choice/design. It's not that Linux doesn't do routing by default, rather the opposite problem that I didn't want it to route that way, so I purposefully avoided it so I could write the routes the way I wanted them.

WireGuard is still under active development, yes. It's considered stable though (despite the ass covering warnings in the code), and is used in production around the world now by data centres, VPN providers and Fortune 500 businesses. I get my full line speed of just under 400Mbps barely breaking single digit CPU usage with WireGuard. It's absolutely wonderful software, and Jason (Donenfeld, the developer) is a total genius.

As for my motivations, it's not so much that pfSense was 'too easy' (though it is in a way, as a GUI can be restrictive if you understand how to use the shell/terminal). I actually love the BSDs, on which pfSense and OPNsense are built, and use them at home myself. However WireGuard isn't available as a kernel module (yet, one is in the works) for *BSD and that comes with a small performance penalty. Furthermore, and more importantly, pfSense has no WireGuard packages available and they don't plan to for some time. OPNsense has a testing package in the 'dev' repo for WireGuard, but it doesn't work on FreeBSD properly. The package itself is sound, but there is apparently a bug upstream (at FreeBSD) when using WireGuard on UFS. Once that's worked out in theory OPNsense is a plug and play possibility if I get bored of Linux (ha!). In reality though Linux has everything I need in this case, especially Arch. It's close to upstream (i.e. fast bug fixes and security updates), it's stable, it's extremely light (100 or so MB RAM usage) and it's infinitely configurable because I can access the shell and a huge software repo directly - whereas *sense limits you to their GUI and their few packages.

A direct example of this would be encrypted DNS. On pfSense you can only use Unbound, only for DNS over TLS, and every lookup uses a new TLS session (which means it's slowed down by nature). On OPNsense they have a dnscrypt-proxy plugin, but it's a case of install here, go to that screen, tick this box, go to another screen, set these options, go to a third screen.... etc. On Linux I just SSH into the box, edit the dnscrypt-proxy.toml file exactly how I want it to work, enable the service and I'm done. I'm glad you enjoyed reading the thread, it was an exercise in frustration for me in some ways. It's hard to learn stuff this complex on your own, especially under the pressure of having a non-functional home network and two toddlers climbing over you as you try to understand policy based routing haha. It took me almost a week all in, but I've gone from a casual Linux user to someone who can install Arch from scratch, code up a router and set up multiple inter-connected networks off the top of my head. It was worth it, and there's a definite sense of accomplishment. Nothing off the shelf would have sufficed (1Gbps VPN throughput over WireGuard/IPSec/OpenVPN, plus total customisation). The EdgeRouter 4 came close on the customisable front - as did Mikrotik stuff. But none have the power to route VPN this fast unless you pay a LOT of money, and even then they're limited. The ER4 maxes out at a few 100 Mbps over VPN for example.

 

[Rainmaker]

If it makes you fee any better, the blame goes both ways... I was going to keep it simple, an APU with 1-2 FTTC connection(s) and some basic OpenVPN routing with an exception to route the local public services normally/via Cloudfare. Then instead of a second FTTC connection to load balance, BT kindly enabled g.fast. I say kindly, it’s turned into a needlessly complicated and drawn out ball ache that’s lasted weeks so far.

Obviously g.fast goes beyond the APU’s single OVPN tunnel speed limit. I’m now into this for a HeX, ER-X, the APU and have a sore i5 6400/Z270 and a bunch of 2T intel NIC’s ready to rock once BT finish screwing with my install. If I had any sense i’d have just virtualised pfsense and stuck to OpenVPN on fast cores, sadly reboots of a virtual environment for updates/upgrades aren’t my idea of fun on consumer (read non certified) hardware as downtime isn’t fun. WireGuard is amazingly quick, but i’m slightly concerned by the bleeding edge nature of the project, despite it’s base.

Ha! We are alike, you and I. Believe it or not I still miss that little APU, it's a nice machine and has personality. I thought so, anyway. The HeX and ER are both things I looked at (I'm familiar with VyOS and have a UAP AC PRO so it was a natural progression). The ER4 would have been ideal but it can't do VPN fast enough for my needs unfortunately.

I wouldn't worry about WireGuard in all honesty. As I said to Steve above, it's considered stable by its developer now and is used around the world in production. The big scary warnings in the code are just ass covering (in the dev's own words) and it's close to inclusion in the mainline kernel now. There have also been independent security audits and WireGuard passed no problems. It's very stable (more so than OpenVPN in my experience) and reliable now. Give it a whirl. I'm actually very curious as to what the APU could push over WireGuard if you'd be so kind as to let me know one day.

I'm very pleased to hear that BT have gotten around to enabling g.fast for you. It's a shame there have been 'teething troubles' but you'll get a nicer connection out of it in the end. I recall from our emails that you were 'stuck' on a fairly slow connection so this will open up some nice possibilities for you once BT get their finger out. Your i5 6400 will scale well beyond gigabit (as you know) so you're set now. What OS are you planning on using? Sticking with pfSense?

 

[Avalon]

Ha! We are alike, you and I. Believe it or not I still miss that little APU, it's a nice machine and has personality. I thought so, anyway. The HeX and ER are both things I looked at (I'm familiar with VyOS and have a UAP AC PRO so it was a natural progression). The ER4 would have been ideal but it can't do VPN fast enough for my needs unfortunately.

I wouldn't worry about WireGuard in all honesty. As I said to Steve above, it's considered stable by its developer now and is used around the world in production. The big scary warnings in the code are just ass covering (in the dev's own words) and it's close to inclusion in the mainline kernel now. There have also been independent security audits and WireGuard passed no problems. It's very stable (more so than OpenVPN in my experience) and reliable now. Give it a whirl. I'm actually very curious as to what the APU could push over WireGuard if you'd be so kind as to let me know one day.

I'm very pleased to hear that BT have gotten around to enabling g.fast for you. It's a shame there have been 'teething troubles' but you'll get a nicer connection out of it in the end. I recall from our emails that you were 'stuck' on a fairly slow connection so this will open up some nice possibilities for you once BT get their finger out. Your i5 6400 will scale well beyond gigabit (as you know) so you're set now. What OS are you planning on using? Sticking with pfSense?

Similar? I did a double take when I looked at your network diagram as it's that similar to my own (you have an AC Pro, i'm still on an original AP). Connection wise I was on FTTC with 80/20, but was looking to load balance two connections with OVPN tunnels for each and and bypass for certain local servers. For 'things' that benefit from a faster connection I rent from OVH/Hetzner/Myloc etc. it's hard to argue with €2.99/m for 10Gb of connectivity, but scaling up adds up quite quickly and peering isn't that great on Hetzner Cloud, so Cloudfare is required.

When I ordered g.fast, it was on the basis of 295/45 being achievable and a minimum of 230/30, I had the HeX before the APU but figured i'd give the ER-X a shot (very similar hardware to HeX) and see how IPSec hardware offload lived up to claims. Then I spotted https://chefkochblog.wordpress.com/2018/05/28/wireguard-already-outperforms-openvpn/ and wondered if the APU may have some life left in it. Official pfsense support isn't happening any time soon from what I read, so it's back to the command line. The 6400 was simply a 'something is going to bleeping well do near line speed VPN one way or another, or else!' moment. I looked at moving my bare metal Ubuntu server (Xeon 2630L) over to Proxmox to host Ubuntu and pfsense, but it seems offload is broken on intel NIC's, so ESXi may make more sense (and is more commercial), it's either that or shove a VM on the UnRAID box. Anyway nothing's happening till BT come back on Friday to resolve my line issue as i'm pulling a solid 140Mbit down today and that's not cool.

 

[Rainmaker]

Similar? I did a double take when I looked at your network diagram as it's that similar to my own (you have an AC Pro, i'm still on an original AP). Connection wise I was on FTTC with 80/20, but was looking to load balance two connections with OVPN tunnels for each and and bypass for certain local servers. For 'things' that benefit from a faster connection I rent from OVH/Hetzner/Myloc etc. it's hard to argue with €2.99/m for 10Gb of connectivity, but scaling up adds up quite quickly and peering isn't that great on Hetzner Cloud, so Cloudfare is required.

When I ordered g.fast, it was on the basis of 295/45 being achievable and a minimum of 230/30, I had the HeX before the APU but figured i'd give the ER-X a shot (very similar hardware to HeX) and see how IPSec hardware offload lived up to claims. Then I spotted https://chefkochblog.wordpress.com/2018/05/28/wireguard-already-outperforms-openvpn/ and wondered if the APU may have some life left in it. Official pfsense support isn't happening any time soon from what I read, so it's back to the command line. The 6400 was simply a 'something is going to bleeping well do near line speed VPN one way or another, or else!' moment. I looked at moving my bare metal Ubuntu server (Xeon 2630L) over to Proxmox to host Ubuntu and pfsense, but it seems offload is broken on intel NIC's, so ESXi may make more sense (and is more commercial), it's either that or shove a VM on the UnRAID box. Anyway nothing's happening till BT come back on Friday to resolve my line issue as i'm pulling a solid 140Mbit down today and that's not cool.

For the APU you'd be best served with a simple CLI install of Linux (pick your poison) and just add in the WireGuard DKMS module. Being multicore aware it'll use all four cores on the APU nicely, but at 1GHz each who knows what the throughput would end up capped at. It'll do gigabit wire speed on Linux no problems just for NAT/netfilter/routing so if you ever got around to it I'd be interested to hear. It'd be ironic if things went full circle and I could have kept it after all (four interfaces, gigabit WAN, decent WireGuard throughput), ha! As you know it can't do much more than 150 to 200Mbps (iirc) openvpn though, thanks to it being single-threaded.

I agree about the 10Gbps connectivity; I do similar but I have to pay slightly more for a less... concerned... provider, who ignores shouty letters from certain groups. I really hope BT (or someone, anyone!) pulls their finger out where I live. VM have the monopoly and they're oversubscribed as usual. It's not too bad this last 6 months but still not great. It's that or 30Mbps on FTTC unfortunately. It's a shame you're having issues with it, but I'm sure they'll sort it. While frustrating, at least 140 is usable in the interim. Hopefully they have you back towards 300Mbps sooner rather than later!

Do you still have my email? Drop me a line in Trust if not, I'd be interested in hearing how you get on all round.

 

[Avalon]

For the APU you'd be best served with a simple CLI install of Linux (pick your poison) and just add in the WireGuard DKMS module. Being multicore aware it'll use all four cores on the APU nicely, but at 1GHz each who knows what the throughput would end up capped at. It'll do gigabit wire speed on Linux no problems just for NAT/netfilter/routing so if you ever got around to it I'd be interested to hear. It'd be ironic if things went full circle and I could have kept it after all (four interfaces, gigabit WAN, decent WireGuard throughput), ha! As you know it can't do much more than 150 to 200Mbps (iirc) openvpn though, thanks to it being single-threaded.

I agree about the 10Gbps connectivity; I do similar but I have to pay slightly more for a less... concerned... provider, who ignores shouty letters from certain groups. I really hope BT (or someone, anyone!) pulls their finger out where I live. VM have the monopoly and they're oversubscribed as usual. It's not too bad this last 6 months but still not great. It's that or 30Mbps on FTTC unfortunately. It's a shame you're having issues with it, but I'm sure they'll sort it. While frustrating, at least 140 is usable in the interim. Hopefully they have you back towards 300Mbps sooner rather than later!

Do you still have my email? Drop me a line in Trust if not, I'd be interested in hearing how you get on all round.

BT will sort it one way or another, though my story about the time Gavin Patterson made me (and others) redundant, nearly got knocked out and was reduced to tears then had to come back later to finish the job possibly carries less weight than it used to.

email? I spam filtered you ages ago Just kidding. From what I read, 600Mbit+ is possible in synthetic testing: https://teklager.se/en/knowledge-base/apu2-vpn-performance/ Not quite sure how I feel about that after buying an ER-L and the i5 build, but half the fun is playing. I'll send an email over later on.

 

[Rainmaker]

Sorry to bump the thread, but again for the benefit of any future searchers/finders, my setup is now on GitHub. I'll format the README nicely over the next day or two, but the basics are all there now.