We have a pfSense thread, but nothing for other stuff - Open/Free/Ghost/Hardened/Net/DragonFly BSD, IPFire, OPNSense, Untangled, Sophos, and whatever else exists for network appliances atm. In all honesty, I think in retrospect an 'Open source router thread' or similar would have been tidier. Feel free to use this thread for your related stories, experiments and findings, as well as any responses to my OP.
As some of you will know by now, I have a home network consisting of VM 550 Internet, an x86 box running Arch Linux as router (netctl, dhcpd4, shorewall, etc), with two local hard separated subnets (LAN and DMZ for CCTV, NAS, Docker and about a dozen servers). Each subnet has its own core switch with branches thereof around the home as necessary (wired).
The x86 Linux router is a repurposed Dell Optiplex 7010 off eBay for £150:
The core switch on my DMZ is PoE due to the HikVision 4K CCTV cameras and Unifi UAP AC PRO. Lately, during my weekly network upgrade/patch day (you know your network's getting serious when you have to schedule mainteance lol) the core DMZ switch has been freezing up and refusing to pass any traffic until it's hard power cycled. It's a (relatively) cheap TP-Link 8 port PoE job and has lasted a couple of years. Time to replace.
I ordered a Netgear ProSAFE GS516TP Smart Pro (16 port, 8x PoE af), which was delivered yesterday. I figured since the network needed a planned outage to replace the switch and reboot the servers etc, I may as well get around to experimenting with OpenBSD as replacement for Linux on my gateway/edge router.
I have long admired OpenBSD and pf (having used pfSense and OPNSense over the years), but had also often read about how - despite being the 'original' and 'best' pf - OpenBSD is single threaded and supposedly too slow for modern fast Internet connections. That 'too slow' claim was, in hindsight, mostly what I read about from pfSense sources (forums, Reddit).
Meanwhile OpenBSD themselves (and Calomel, whom I admire and use as a reference often) said that, actually, OpenBSD is now multi threaded in a lot of ways. It has an mp kernel and its pf will - while still being single threaded for now - be so light that even high speed connections will be bottlenecked by the NIC long before the CPU used is a problem. I decided to give it a try and see for myself! That's where our story begins...
I obviously wasn't going to mess around with this on an in-production machine, so I dug out an older mini-ITX USFF machine I bought a couple of years ago:
I flashed the newly released OpenBSD 6.7 to a USB (sudo dd if=install67.fs of=/dev/sdc) and booted up. The install was literally a case of choosing (I)nstall, next, next, next, done, reboot. I configured the WAN and LAN interfaces per the (famously good) OpenBSD man pages:
/etc/hostname.em0
/etc/hostname.em1
and set DHCP to dish out IPs between 10.100.0.50 and 10.100.0.250 (leaving room for servers and wired LAN machines at the beginning of the subnet, and managed switches at the end of it).
/etc/dhcpd.conf
I then set up OpenBSD's famously good firewall, pf, to allow all outgoing traffic and block anything coming in that shouldn't be, while redirecting (port forwarding) essential service ports to the nginx reverse proxy on my NAS to Docker:
/etc/pf.conf
I could also have set up unbound to forward LAN clients to the DNS server on the NAS (AdGuard Home running DoH and DoT with DNSSEC and adblocking), but I simply set DHCP to point clients to the NAS directly for DNS. This means I can have the gateway as a simple inaccessible forwarding device (as a router was intended to be), with the only access it accepts from WAN or LAN being over ssh with a key (no passwords).
After issuing pfctl -f /etc/pf.conf to refresh the ruleset, the network came up, established connections immediately, and it was set. Literally 10 minutes work (including SSHing over the configs I had pre-written). I put a MacBook Pro on the LAN interface and plugged the WAN into my existing LAN switch (behind the 'real' Arch Linux router) for testing.
The OpenBSD box reports it's using 29 MB of RAM with all interfaces up, pf ruleset loaded, and dhcp + nat established. For comparison, Linux with the same config hovers around 250 MB!
An iperf3 test to a 10Gb server on the wider Internet returned an immediate and solid 550Mbps downstream (my ISP max). Since OpenBSD now has an in-kernel WireGuard module(!) I loaded up one of my client configs for a strong, privacy focused VPN provider I trust. I re-did the iperf3 tests, and got a solid 530Mbps down.
During this, htop reported the CPU load as 6% on core 1 and 3% on core 2. So much for OpenBSD being too slow for high speed networks! Scaled linearly, my crappy old Pentium machine could pull almost 10Gb before hitting CPU limits. That'll do for some years to come (unfortunately - I wish we had 10Gb fibre here like our European and Scandanavian cousins).
So, success, right?! Well, not quite...
I then, stupidly, decided to complete the setup by updating the machine's UEFI/BIOS. It hadn't been patched in over 2 years, and with Intel's laughable security record lately in addition to this being public-facing core network infrastructure, I decided to upgrade. ASRock had released a BIOS containing new microcode as recently as this month!
This is the part where I mention that all this happened on Tuesday evening. I hadn't slept since Sunday... Do you see what's coming?
The BIOS flashed without a hitch, but upon reboot OpenBSD informed me the eeprom checksum for em1 was now invalid and thus the interface wouldn't be coming online. Merde. This is fairly easy to fix in Linux, but I CBA by this point. Easier to flash back to an older BIOS and sort the problem another day. Rolling back 1 version didn't help. I had forgotten to make a note of the existing (working) BIOS version so I quickly grabbed all versions from ASRock's site. Back 2 versions also didn't work.
At this stage I figure the box hasn't been updated in 2+ years, so if I flash the *earliest* BIOS and work forwards, that'll be faster. So I duly flash the first available version. No boot. Oops... CPU mircrocode for Kaby Lake wasn't added to the UEFI until version 1.30. I now own a (soft) brick.
So, eBay to the rescue (OcUK don't sell CPUs this old) and I paid £25 for an ancient and crappy Celeron (2 core, 2.4GHz 43W TDP iirc). That Celeron is supported by the installed BIOS, so once it arrives I can quickly swap CPUs, flash to the latest BIOS, put the Pentium back in place and fix the NIC. Eejit! Don't do critical work while two-day-tired, folks! At least my reckless tired brain only cost me £25 this time - I've had worse!
Regardless, that's just a funny (in hindsight) story. It's easily fixable and wouldn't have happened if I hadn't been a plonker. It's not OpenBSD's fault nor anything to do with it. Once I fix the BIOS/UEFI this box is a very capable, eminently secure and easy to work with appliance.
I'm actually going to fix the machine as planned, but then just put Artix Linux (Arch without systemd) on it for my kids to play with. The experiment was so successful and OpenBSD is such an absolute joy to work with, that I'll instead schedule the last patch day of the month as network update day - installing the new switch, refreshing the network topology and having OpenBSD overwrite the Linux OS on my existing (more powerful Core i7) x86 machine.
So there we have it. A little story about trying OpenBSD, with a happy ending (and some caveats about sleep!). I actually found after a few years of using pfSense et al. that I was starting to become hamstrung by the GUI - it was just slowing me down. That's nothing special, in fact I've ready that from *nix admins for all the 20 years I've been involved with open source. At some point as you become experienced, it's easier just to fire up VIM (or whatever) and write the config in plaintext yourself. Simple, elegant and easier to find/read/debug as required. GUIs just get in the way!
That's my networking and open source adventure for the week. I've made my network massively more secure, POSIX compliant (goodbye systemd) and easier to work with. What about you?...
As some of you will know by now, I have a home network consisting of VM 550 Internet, an x86 box running Arch Linux as router (netctl, dhcpd4, shorewall, etc), with two local hard separated subnets (LAN and DMZ for CCTV, NAS, Docker and about a dozen servers). Each subnet has its own core switch with branches thereof around the home as necessary (wired).
The x86 Linux router is a repurposed Dell Optiplex 7010 off eBay for £150:
Code:
Intel Core i7 3770 4c8t
8GB RAM
120GB SSD
Intel Pro 1000 VT quad port server NICThe core switch on my DMZ is PoE due to the HikVision 4K CCTV cameras and Unifi UAP AC PRO. Lately, during my weekly network upgrade/patch day (you know your network's getting serious when you have to schedule mainteance lol) the core DMZ switch has been freezing up and refusing to pass any traffic until it's hard power cycled. It's a (relatively) cheap TP-Link 8 port PoE job and has lasted a couple of years. Time to replace.
I ordered a Netgear ProSAFE GS516TP Smart Pro (16 port, 8x PoE af), which was delivered yesterday. I figured since the network needed a planned outage to replace the switch and reboot the servers etc, I may as well get around to experimenting with OpenBSD as replacement for Linux on my gateway/edge router.
I have long admired OpenBSD and pf (having used pfSense and OPNSense over the years), but had also often read about how - despite being the 'original' and 'best' pf - OpenBSD is single threaded and supposedly too slow for modern fast Internet connections. That 'too slow' claim was, in hindsight, mostly what I read about from pfSense sources (forums, Reddit).
Meanwhile OpenBSD themselves (and Calomel, whom I admire and use as a reference often) said that, actually, OpenBSD is now multi threaded in a lot of ways. It has an mp kernel and its pf will - while still being single threaded for now - be so light that even high speed connections will be bottlenecked by the NIC long before the CPU used is a problem. I decided to give it a try and see for myself! That's where our story begins...
I obviously wasn't going to mess around with this on an in-production machine, so I dug out an older mini-ITX USFF machine I bought a couple of years ago:
Code:
ASRock Industrial IMB-191
Intel (Kaby Lake) Pentium G4560 2c4t @ 3.5GHz
8GB RAM
120GB SSD
2x Intel NIC (I210 and I219-V).I flashed the newly released OpenBSD 6.7 to a USB (sudo dd if=install67.fs of=/dev/sdc) and booted up. The install was literally a case of choosing (I)nstall, next, next, next, done, reboot. I configured the WAN and LAN interfaces per the (famously good) OpenBSD man pages:
/etc/hostname.em0
Code:
dhcp/etc/hostname.em1
Code:
inet 10.100.0.1 255.255.255.0 10.100.0.255and set DHCP to dish out IPs between 10.100.0.50 and 10.100.0.250 (leaving room for servers and wired LAN machines at the beginning of the subnet, and managed switches at the end of it).
/etc/dhcpd.conf
Code:
subnet 10.100.0.0 netmask 255.255.255.0 {
option routers 10.100.0.1;
option domain-name-servers 10.100.0.5;
range 10.100.0.50 10.100.0.250;
host Rainmaker-PC {
hardware ethernet 12:34:A5:67:8B:C9;#Intel I210 on Strix TRX40
fixed-address 10.100.0.2;
option host-name "Rainmaker-PC";
}
host etc etc...
}I then set up OpenBSD's famously good firewall, pf, to allow all outgoing traffic and block anything coming in that shouldn't be, while redirecting (port forwarding) essential service ports to the nginx reverse proxy on my NAS to Docker:
/etc/pf.conf
Code:
### Interfaces
wan = "em0"
lan = "em1"
lan_net = "10.100.0.0/24"
### Host macros
nas = "10.100.0.5"
### Port macros
nas_tcp = "{ 21, 80, 443, 8096, 32400 }"
adguard_tcp = "{ 444, 853, 8888 }"
adguard_udp = "{ 444, 853 }"
### Tables
table <martians> { 0.0.0.0/8 10.0.0.0/8 127.0.0.0/8 169.254.0.0/16 \
172.16.0.0/12 192.0.0.0/24 192.0.2.0/24 224.0.0.0/3 \
192.168.0.0/16 198.18.0.0/15 198.51.100.0/24 \
203.0.113.0/24 }
### Global policy
set block-policy drop
set loginterface egress
set skip on lo
match in all scrub (no-df random-id max-mss 1440)
match out on egress inet from !(egress:network) to any nat-to (egress:0)
block in quick on egress from <martians> to any
block return out quick on egress from any to <martians>
antispoof quick for { egress $lan }
### Rules
block log all
pass out quick from self
pass in on $lan
pass out on $lan
pass inet proto icmp all icmp-type { echoreq unreach }
#
# Allow SSH to router
#
pass in log on egress inet proto tcp from any to (egress) port 22
#
# DNAT
#
pass in on egress inet proto tcp from any to (egress) port $nas_tcp rdr-to $nas
pass in on egress inet proto tcp from any to (egress) port $adguard_tcp rdr-to $nas
pass in on egress inet proto udp from any to (egress) port $adguard_udp rdr-to $nasI could also have set up unbound to forward LAN clients to the DNS server on the NAS (AdGuard Home running DoH and DoT with DNSSEC and adblocking), but I simply set DHCP to point clients to the NAS directly for DNS. This means I can have the gateway as a simple inaccessible forwarding device (as a router was intended to be), with the only access it accepts from WAN or LAN being over ssh with a key (no passwords).
After issuing pfctl -f /etc/pf.conf to refresh the ruleset, the network came up, established connections immediately, and it was set. Literally 10 minutes work (including SSHing over the configs I had pre-written). I put a MacBook Pro on the LAN interface and plugged the WAN into my existing LAN switch (behind the 'real' Arch Linux router) for testing.
The OpenBSD box reports it's using 29 MB of RAM with all interfaces up, pf ruleset loaded, and dhcp + nat established. For comparison, Linux with the same config hovers around 250 MB!
An iperf3 test to a 10Gb server on the wider Internet returned an immediate and solid 550Mbps downstream (my ISP max). Since OpenBSD now has an in-kernel WireGuard module(!) I loaded up one of my client configs for a strong, privacy focused VPN provider I trust. I re-did the iperf3 tests, and got a solid 530Mbps down.
During this, htop reported the CPU load as 6% on core 1 and 3% on core 2. So much for OpenBSD being too slow for high speed networks! Scaled linearly, my crappy old Pentium machine could pull almost 10Gb before hitting CPU limits. That'll do for some years to come (unfortunately - I wish we had 10Gb fibre here like our European and Scandanavian cousins).
So, success, right?! Well, not quite...
I then, stupidly, decided to complete the setup by updating the machine's UEFI/BIOS. It hadn't been patched in over 2 years, and with Intel's laughable security record lately in addition to this being public-facing core network infrastructure, I decided to upgrade. ASRock had released a BIOS containing new microcode as recently as this month!
This is the part where I mention that all this happened on Tuesday evening. I hadn't slept since Sunday... Do you see what's coming?
The BIOS flashed without a hitch, but upon reboot OpenBSD informed me the eeprom checksum for em1 was now invalid and thus the interface wouldn't be coming online. Merde. This is fairly easy to fix in Linux, but I CBA by this point. Easier to flash back to an older BIOS and sort the problem another day. Rolling back 1 version didn't help. I had forgotten to make a note of the existing (working) BIOS version so I quickly grabbed all versions from ASRock's site. Back 2 versions also didn't work.
At this stage I figure the box hasn't been updated in 2+ years, so if I flash the *earliest* BIOS and work forwards, that'll be faster. So I duly flash the first available version. No boot. Oops... CPU mircrocode for Kaby Lake wasn't added to the UEFI until version 1.30. I now own a (soft) brick.
So, eBay to the rescue (OcUK don't sell CPUs this old) and I paid £25 for an ancient and crappy Celeron (2 core, 2.4GHz 43W TDP iirc). That Celeron is supported by the installed BIOS, so once it arrives I can quickly swap CPUs, flash to the latest BIOS, put the Pentium back in place and fix the NIC. Eejit! Don't do critical work while two-day-tired, folks! At least my reckless tired brain only cost me £25 this time - I've had worse!
Regardless, that's just a funny (in hindsight) story. It's easily fixable and wouldn't have happened if I hadn't been a plonker. It's not OpenBSD's fault nor anything to do with it. Once I fix the BIOS/UEFI this box is a very capable, eminently secure and easy to work with appliance.
I'm actually going to fix the machine as planned, but then just put Artix Linux (Arch without systemd) on it for my kids to play with. The experiment was so successful and OpenBSD is such an absolute joy to work with, that I'll instead schedule the last patch day of the month as network update day - installing the new switch, refreshing the network topology and having OpenBSD overwrite the Linux OS on my existing (more powerful Core i7) x86 machine.
So there we have it. A little story about trying OpenBSD, with a happy ending (and some caveats about sleep!). I actually found after a few years of using pfSense et al. that I was starting to become hamstrung by the GUI - it was just slowing me down. That's nothing special, in fact I've ready that from *nix admins for all the 20 years I've been involved with open source. At some point as you become experienced, it's easier just to fire up VIM (or whatever) and write the config in plaintext yourself. Simple, elegant and easier to find/read/debug as required. GUIs just get in the way!
That's my networking and open source adventure for the week. I've made my network massively more secure, POSIX compliant (goodbye systemd) and easier to work with. What about you?...
Last edited:
