pfSense halving Virgin fibre connection speed

Vauxmad · 16 Nov 2019 at 17:39

I'm hoping someone with more pfSense expericence can help me with an issue im facing.

I've got a Dell R210ii running pfSense 2.4.4-RELEASE-p3. Ive been using this with my plusnet Fibre connection with no issues for the past few years.

I've just had 100mb virgin fibre installed. When testing the superhub 3 in router i get 110mb on speedtest.net. When i put it into modem mode and connecting through pfSense my speed drops to ~55mb.

I've never had any problems with pfsense before so need some help with diagnosing the issue. Ive looked at the gateway log and can see some packet loss.

Code:

WAN Interface (wan, bge0)
Status
up
DHCP
up     Relinquish Lease
MAC Address
00:0a:f7:44:5c:bc
IPv4 Address
##.##.##.##
Subnet mask IPv4
255.255.240.0
Gateway IPv4
##.##.##.##
IPv6 Link Local
fe80::2ad2:44ff:fe4e:4e18%bge0
DNS servers
127.0.0.1
192.168.0.235
MTU
1500
Media
100baseTX <full-duplex>
In/out packets
15284664/7300870 (19.39 GiB/971.09 MiB)
In/out packets (pass)
15284664/7300870 (19.39 GiB/971.09 MiB)
In/out packets (block)
100593/107 (3.20 MiB/19 KiB)
In/out errors
325747/0

Code:

Nov 16 16:49:56    dpinger        send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr ##.##.##.## bind_addr ##.##.##.## identifier "WAN_DHCP "
Nov 16 16:50:00    dpinger        send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr ##.##.##.## bind_addr ##.##.##.## identifier "WAN_DHCP "
Nov 16 16:50:52    dpinger        WAN_DHCP ##.##.##.##: Alarm latency 9457us stddev 4338us loss 21%
Nov 16 16:51:51    dpinger        WAN_DHCP ##.##.##.##: Clear latency 11589us stddev 8548us loss 9%
Nov 16 17:22:37    dpinger        send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr ##.##.##.## bind_addr ##.##.##.## identifier "WAN_DHCP "
Nov 16 17:22:41    dpinger        send_interval 500ms loss_interval 2000ms time_period 60000ms report_interval 0ms data_len 0 alert_interval 1000ms latency_alarm 500ms loss_alarm 20% dest_addr ##.##.##.## bind_addr ##.##.##.## identifier "WAN_DHCP "
Nov 16 17:23:12    dpinger        WAN_DHCP ##.##.##.##: Alarm latency 9723us stddev 4958us loss 21%
Nov 16 17:23:42    dpinger        WAN_DHCP ##.##.##.##: Clear latency 22160us stddev 19379us loss 17%

Caged · 16 Nov 2019 at 18:25

Leave the Superhub in router mode and double-NAT pfSense as a test - you might be diagnosing the wrong problem

Steveocee · 16 Nov 2019 at 18:38

What CPU are you using in your Dell? What network card (if any)? Are you using any sort of QoS or packet inspection?

Try turning off all traffic management first, it might be you have queues set up which will be limiting speeds.

sideways14a · 16 Nov 2019 at 19:04

Pfsense is dependant on cpu performance (or VM performance)
I went from 39Mb/s with a stock plusnet (and VM before it router) to 48Mb/s with pfsense running on a 24thread server (VM) - it sped up each time i gave it more resources... plus tweeks to the vdsl link.
Mine now runs with something like 6 threads, 24gb of ram and 3 dedicated 1gb network cards for its vlans and all that.

Vauxmad · 16 Nov 2019 at 22:36

Caged said:
Leave the Superhub in router mode and double-NAT pfSense as a test - you might be diagnosing the wrong problem

just tried this, laptop connection to super hub WiFi gets ~110mb, pc connected to pfsense and only getting ~55mb

Steveocee said:
What CPU are you using in your Dell? What network card (if any)? Are you using any sort of QoS or packet inspection?

Try turning off all traffic management first, it might be you have queues set up which will be limiting speeds.

cpu is an i3 2100 - pfsense showing minimal cpu useage, I’ve got a quad gig intel nic + the 2 onboard nics and get the same behaviour on either onboard or the intel nic. For some reason pfsense is only connecting at 100baseTX if I try 1000baseTX I get no connection

sideways14a said:
Pfsense is dependant on cpu performance (or VM performance)
I went from 39Mb/s with a stock plusnet (and VM before it router) to 48Mb/s with pfsense running on a 24thread server (VM) - it sped up each time i gave it more resources... plus tweeks to the vdsl link.
Mine now runs with something like 6 threads, 24gb of ram and 3 dedicated 1gb network cards for its vlans and all that.

Pfsense shows very little cpu utilisation and 7% ram (running 8gb)

sideways14a · 16 Nov 2019 at 23:19

i3 isnt particularly great and the problem with the nics is a big one.

I have little problem with mine in either esxi or hyper V, if your not using one of them ... well then maybe you should move the build over to see if it helps.
but i definitely saw scaling when moving up in resources, my old dual core hp micro server was a bottleneck at sub 40mb as well.

Rainmaker · 17 Nov 2019 at 04:57

You appear to be linked at 100Mbps not 1Gbps on the NIC:

Code:

100baseTX <full-duplex>

Check the cable and port, and check your negotiation settings.

Also, is your WAN interface directly attached/bound to the individual NIC port, or do you have some weird bridge thing going on? Running a software bridge will kill your speeds like a lead brick...

Code:

WAN Interface (wan, bge0)

pfSense/FreeBSD is good for at least a couple of Gbps on decent hardware. Sorting the above should make a big difference.

Vauxmad · 17 Nov 2019 at 10:35

So after wasting the best part of a day troubleshooting i missed one of the first basic tests - the homehub is in my office and the pfsense router is up in a rack in the loft. I had swapped both patch leads at either end but not tested the structured cabling. Swapped data ports (luckily ran extra drops!) and now getting full speed.

I can go back to looking for a cheap Xeon E3 now to replace the i3

Thanks to all who replied!

Steveocee · 17 Nov 2019 at 11:51

Vauxmad said:
So after wasting the best part of a day troubleshooting i missed one of the first basic tests - the homehub is in my office and the pfsense router is up in a rack in the loft. I had swapped both patch leads at either end but not tested the structured cabling. Swapped data ports (luckily ran extra drops!) and now getting full speed.

I can go back to looking for a cheap Xeon E3 now to replace the i3

Thanks to all who replied!

E3-1265L are the ones people suggest due to lower power but I've been running a 1270 for over 2 years now. Cooling hasn't been an issue at all either through this summer (rack is in loft) with a hefty Poe switch with it as well.

Caged · 17 Nov 2019 at 12:52

I can't get on board with the idea that an i3 is bottlenecking a 100Mbps connection or that somehow you need 6 cores and 24GB of RAM for an FTTC link - at that point you're trying to patch up a significant problem by throwing resources at it. The little £130 Netgate embedded boxes can happily push a few hundred Mbps through them.

Vauxmad · 17 Nov 2019 at 15:34

Caged said:
I can't get on board with the idea that an i3 is bottlenecking a 100Mbps connection or that somehow you need 6 cores and 24GB of RAM for an FTTC link - at that point you're trying to patch up a significant problem by throwing resources at it. The little £130 Netgate embedded boxes can happily push a few hundred Mbps through them.

The main reason i'm looking for the Xeon is for AES-NI ready for when netgate make it a requirement (i know that's been removed for 2.5 but still planning ahead). I got the r210ii for 70quid and dont plan on spending more than 15quid on a new cpu

I dont think Ive ever seen my memory use in double figures and im running 8gig

sideways14a · 17 Nov 2019 at 15:45

No you dont need 6 cores and 24 Mb but thats what i gave it because i have resources a plenty - and there was a significant speed up from the dual core i ran before it. Quite surprised when i moved it and got circa 10Mb/s improvement (with tweaks though)
There are also the hardware encryption options that seem to make a different if your cpu can handle it.

Of course if you are running 100Mb its always going to be a problem.
Edited to say i saw a lot more cpu usage when i turned on all the fancy security and packet inspection systems like snort and all that.

Avalon · 17 Nov 2019 at 21:38

Anyone suggesting an i3 would be a bottleneck for a basic 100Mbit connection under pfsense shouldn’t be offering advice and that’s about as polite as I can manage to be. Also virtualising pfsense, chucking additional plugins into the mix and then comparing the resource allocation to bare metal installs directly isn’t improving matters, yes in a homelab it’s acceptable practice to run that way, but why would you compare it to OP’s situation and if you feel you must, without context?

So let’s start with the i3-2100, my first R210-II came with one too, it handles 350/50 effortlessly and was sub 50w. The 1270v1 that replaced it is also sub 50w. Heck even the 1Ghz quad core GX-412TC can do 350/50 but did crap out at 95Mbit of OpenVPN per tunnel interface, so the CPU is not the issue here.

Also can we please stop perpetuating the nonsense about low TDP intel chips magically saving power in servers essentially running router distributions at what is usually ‘idle’? Intel power-gating doesn’t work like that in the generation(s) being discussed here, idle i7’s and Xeon’s of this generation use barely any more power than an L equivalent or an i3/5. Lower TDP chips are for scenario’s where cooling capacity is limited, the R210-II is a 50w machine with a 1270v1 in it, it’s basically the same with a i3 2100 in it, the difference is where it actually does stuff (or AES-NI for hardware encryption) but it does it quicker and goes back to idle.

Rabtech · 17 Nov 2019 at 22:25

I ran my previous Virgin 350/35 on an AM1 5350 (bare metal) and that ran at about 40% CPU at full whack (multi threaded and single threaded downloads) with no IDS/IPS. You only generally need raw horsepower in pfsense if you are running suricata or snort (IDS/IPS).

OP - are you sure you aren't suffering from one of the network interface related bugs?

If it helps, under System > Advanced > Networking I have 'Hardware Checksum Offloading' unticked (to enable it), 'Hardware TCP Segmentation Offloading' ticked (to disable it) and 'Hardware Large Receive Offloading' ticked (to disable it)

Rabtech · 17 Nov 2019 at 22:38

Avalon said:
Also can we please stop perpetuating the nonsense about low TDP intel chips magically saving power in servers essentially running router distributions at what is usually ‘idle’? Intel power-gating doesn’t work like that in the generation(s) being discussed here, idle i7’s and Xeon’s of this generation use barely any more power than an L equivalent or an i3/5. Lower TDP chips are for scenario’s where cooling capacity is limited, the R210-II is a 50w machine with a 1270v1 in it, it’s basically the same with a i3 2100 in it, the difference is where it actually does stuff (or AES-NI for hardware encryption) but it does it quicker and goes back to idle.

I agree with this, I already had the AM1 so I just used that but I was looking various chinese dual NIC - NUC boxes with i5's and i7's. I concluded it was a waste of time and when 1Gb is available in my area I'll just go for a very high clocked dual core pentium on a mini-itx board. Should be more than good enough to handle the speed with IPS/IDS thrown in.