OPNSense & BT Full Fibre 900 (PPPoE)

ChrisD. · 29 Dec 2022 at 17:58

Dodger said:
I will run the command and share the results.
Networking mode in the QNAP virtual machine setup is ‘Bridged network’.

Is the WAN interface 'external'?

Then have the LAN interface as bridged.

Networking modes of Virtualization Station

Virtualization Station supports three different networking modes for Virtual Switch connections: bridged, External-only and isolat ...

www.qnap.com

Dodger · 29 Dec 2022 at 18:21

I’ve attached some photos to help show my current setup which may help potentially root cause diagnose any potential setup issues, thanks.

8-D71-B410-3-FA5-4444-B267-786-B0932303-D hosted at ImgBB

Image 8-D71-B410-3-FA5-4444-B267-786-B0932303-D hosted in ImgBB

ibb.co

4-DF21-BE0-54-E6-477-D-98-E5-95-E6-E40-E66-A1 hosted at ImgBB

Image 4-DF21-BE0-54-E6-477-D-98-E5-95-E6-E40-E66-A1 hosted in ImgBB

ibb.co

BB311145-09-A4-41-F4-A4-FA-3-B3699154495 hosted at ImgBB

Image BB311145-09-A4-41-F4-A4-FA-3-B3699154495 hosted in ImgBB

ibb.co

DF8-CDD65-820-E-4613-BB34-6-C941-B794-C52 hosted at ImgBB

Image DF8-CDD65-820-E-4613-BB34-6-C941-B794-C52 hosted in ImgBB

ibb.co

ChrisD. · 29 Dec 2022 at 18:24

Dude, clean your screen!

Put adaptor 1 on an external network switch, not bridged mode and see if that improves things. Also, change the MTU of the external switch to 1504 and also enable baby jumbo frames within OPNsense, that may also improve things.

Dodger · 29 Dec 2022 at 18:30

ChrisD. said:
ssh to it and run the command, it should automatically show whatever is using CPU at the top of the list. I don't know if it has top or htop bundled.

Also what networking mode is it? bridged, host etc?

Ran the top command.
From what I could see, the CPU did not go above 5%…hopefully I’ve ran the correct command/looking at the right thing!

F8180981-02-BB-4-B7-E-8-AAE-9-E94-AEDA0-AF2 hosted at ImgBB

Image F8180981-02-BB-4-B7-E-8-AAE-9-E94-AEDA0-AF2 hosted in ImgBB

ibb.co

Rainmaker · 29 Dec 2022 at 18:33

Dodger said:
Is this what you are referring to in relation to using em?

You've made progress since I posted, and tbh this is probably more a virtualisation/Qnap issue rather than anything *BSD based. However, to answer your question: No, different NICs require different drivers, even from the same manufacturer. For example, staying relevant to this case, some Intel NICs use the em driver and some use the igb driver. It's the former which has issues with PPPoE on FreeBSD and related OS (like OPNSense).

If you use such a NIC (eg Intel I350) you will suffer single queue/core issues with maxing speed under PPPoE. Switching to a NIC that uses any other driver (such as the aforementioned igb driver) removes this bug/limitation and no speed issues present, PPPoE or not. You can check which driver you're using by checking the dmesg log: more /var/run/dmesg.boot and scrolling through until you see mention of the network adapter(s) and their driver.

Edit: On *BSD PPPoE is still single-threaded and as such will only use a single encapsulated queue on the NIC. My advice about running Linux based OS for routing/firewalling using PPPoE stands, and your headache will instantly disappear... Try OpenWRT, VyOS or IPFire (no IPv6 or WireGuard in the latter), or even barebones Debian and some text file config magic.

Dodger · 29 Dec 2022 at 18:33

ChrisD. said:
Dude, clean your screen!

Put adaptor 1 on an external network switch, not bridged mode and see if that improves things. Also, change the MTU of the external switch to 1504 and also enable baby jumbo frames within OPNsense, that may also improve things.

Hahaha!!! Once its all setup, i’ll have a office cleanup!

The only option I see in the virtual machine is to select the ‘User mode networking’ option rather than ‘Bridged network’ - is this what you meant by putting adapter 1 on an external network switch?

Thanks.

ChrisD. · 29 Dec 2022 at 18:35

You may have to delete the adaptor and add a new one, QNAP is a bit odd at times with VM settings.

Dodger · 29 Dec 2022 at 19:17

Rainmaker said:
You've made progress since I posted, and tbh this is probably more a virtualisation/Qnap issue rather than anything *BSD based. However, to answer your question: No, different NICs require different drivers, even from the same manufacturer. For example, staying relevant to this case, some Intel NICs use the em driver and some use the igb driver. It's the former which has issues with PPPoE on FreeBSD and related OS (like OPNSense).

If you use such a NIC (eg Intel I350) you will suffer single queue/core issues with maxing speed under PPPoE. Switching to a NIC that uses any other driver (such as the aforementioned igb driver) removes this bug/limitation and no speed issues present, PPPoE or not. You can check which driver you're using by checking the dmesg log: more /var/run/dmesg.boot and scrolling through until you see mention of the network adapter(s) and their driver.

Edit: On *BSD PPPoE is still single-threaded and as such will only use a single encapsulated queue on the NIC. My advice about running Linux based OS for routing/firewalling using PPPoE stands, and your headache will instantly disappear... Try OpenWRT, VyOS or IPFire (no IPv6 or WireGuard in the latter), or even barebones Debian and some text file config magic.

I have looked at dmesg.boot and can’t find anything related to em or ig0…I am using network adapters in a virtualised environment so not sure if this is why?

Yep, take your point regarding BSD…I was going to see how it went with OPNSense and if no success, move over to a Linux based OS firewall/router…

Dodger · 29 Dec 2022 at 19:23

ChrisD. said:
You may have to delete the adaptor and add a new one, QNAP is a bit odd at times with VM settings.

Happy to try this but are you suggesting for the WAN to select the ‘User mode networking’ option?

Dodger · 29 Dec 2022 at 19:50

Also just plugged WAN and LAN into the Intel adapter 3 and 4 and reconfigured the virtual adapters and still same results i.e. slow download speed :-(

Managed to get 897Mbps with a laptop connected directly to the ONT modem…

Dodger · 29 Dec 2022 at 19:57

ChrisD. said:
Dude, clean your screen!

Put adaptor 1 on an external network switch, not bridged mode and see if that improves things. Also, change the MTU of the external switch to 1504 and also enable baby jumbo frames within OPNsense, that may also improve things.

Was not able to select 1504, just 1500 or some other random numbers such as 9000.
I am however able to set a WAN MTU of 1504 if this is what you meant?
Thanks.

Dodger · 29 Dec 2022 at 20:02

For further reference, I have uploaded LAN, WAN and Dashboard Interface photos.

Only weird thing I have noticed is for the WAN, I do not see 10Gbase-T <full-duplex> like I do for the LAN…is this normal when using PPPoE for the WAN?

Thanks.

062-D017-E-53-B7-4898-BDA7-6-FF32856-B1-AF hosted at ImgBB

Image 062-D017-E-53-B7-4898-BDA7-6-FF32856-B1-AF hosted in ImgBB

ibb.co

B963-E242-F496-44-E8-8-B8-F-DB030268-C2-DC hosted at ImgBB

Image B963-E242-F496-44-E8-8-B8-F-DB030268-C2-DC hosted in ImgBB

ibb.co

D0-E1517-C-F56-C-4636-8928-855-A74471991 hosted at ImgBB

Image D0-E1517-C-F56-C-4636-8928-855-A74471991 hosted in ImgBB

ibb.co

Dodger · 3 Jan 2023 at 14:00

Rainmaker said:
You've made progress since I posted, and tbh this is probably more a virtualisation/Qnap issue rather than anything *BSD based. However, to answer your question: No, different NICs require different drivers, even from the same manufacturer. For example, staying relevant to this case, some Intel NICs use the em driver and some use the igb driver. It's the former which has issues with PPPoE on FreeBSD and related OS (like OPNSense).

If you use such a NIC (eg Intel I350) you will suffer single queue/core issues with maxing speed under PPPoE. Switching to a NIC that uses any other driver (such as the aforementioned igb driver) removes this bug/limitation and no speed issues present, PPPoE or not. You can check which driver you're using by checking the dmesg log: more /var/run/dmesg.boot and scrolling through until you see mention of the network adapter(s) and their driver.

Edit: On *BSD PPPoE is still single-threaded and as such will only use a single encapsulated queue on the NIC. My advice about running Linux based OS for routing/firewalling using PPPoE stands, and your headache will instantly disappear... Try OpenWRT, VyOS or IPFire (no IPv6 or WireGuard in the latter), or even barebones Debian and some text file config magic.

I am going to try OpenWRT.

Does anyone know best practices for example, should/can I spin up a virtual machine and run OpenWRT?

Or, is it better to use Docker to run OpenWRT? If so, as a Container or, a Docker host?

Thanks!

robj20 · 3 Jan 2023 at 14:05

Where did you get the 1602p from they seem hard to find, almost like it's end of life.

Dodger · 3 Jan 2023 at 14:07

robj20 said:
Where did you get the 1602p from they seem hard to find, almost like it's end of life.

I managed to get one from Qnap directly. There was a spare 1602P 8GB in a Swindon UK warehouse.

robj20 · 3 Jan 2023 at 14:08

Dodger said:
I managed to get one from Qnap directly. There was a spare 1602 8GB in a Swindon UK warehouse.

They seem really good as an all in one, I fancied it as a router, switch and IP camera recorder, unifi controller.

Dodger · 3 Jan 2023 at 14:10

robj20 said:
They seem really good as an all in one, I fancied it as a router, switch and IP camera recorder, unifi controller.

Yep so far so good, takes abit of time to setup using virtual machines if never done it before (like me)!

Dodger · 21 Jan 2023 at 16:34

Hi, just wanted to post an update to get some further thoughts.

I’ve installed OpenWRT which is running in QNAP’s Virtualization Station.
Then setup a PPPoE connection in OpenWRT.

In a wired test, I’m still getting fluctuating speedtest results, now between 650-850Mbps.

If I plugin my BT Business Smart Hub, I get 930Mbps straight away wired test.

Is the CPU Atom C3558 in the QGD-1602 underpowered which means I won’t be able to achieve a faster download speed?

Any thoughts from anyone please?

Thanks!

Caged · 21 Jan 2023 at 17:03

What's the CPU load on each core when you're running these tests? If they aren't hitting 100% then you aren't exhausting CPU.

Dodger · 21 Jan 2023 at 17:13

Caged said:
What's the CPU load on each core when you're running these tests? If they aren't hitting 100% then you aren't exhausting CPU.

Hi, thanks for the reply.

I’ve installed:

luci-app-statistics

I then ran a speed test an hour or so ago, please find results below:

41609030-F285-49-B3-AE0-C-5-D9-DCA70558-B hosted at ImgBB

Image 41609030-F285-49-B3-AE0-C-5-D9-DCA70558-B hosted in ImgBB

ibb.co

1-B765-D15-EF15-48-F8-84-B8-784347249-B0-B hosted at ImgBB

Image 1-B765-D15-EF15-48-F8-84-B8-784347249-B0-B hosted in ImgBB

ibb.co

C85470-EA-4-C48-400-A-AB6-F-816-AF832-BE9-F hosted at ImgBB

Image C85470-EA-4-C48-400-A-AB6-F-816-AF832-BE9-F hosted in ImgBB

ibb.co

2523-FB29-4535-4-AA9-90-E5-1-B42-DCC9-F711 hosted at ImgBB

Image 2523-FB29-4535-4-AA9-90-E5-1-B42-DCC9-F711 hosted in ImgBB

ibb.co

On the Server, I have 1 CPU core allocated to Docker Unifi, which is running in Container Station.

CPU doesn’t seem to be maxing out based on the attached images from my understanding.

Also to add, in OpenWRT, I have enabled Software Flow Offloading.

Where else could the bottleneck be?

Thanks!

OPNSense & BT Full Fibre 900 (PPPoE)

luci-app-statistics​

luci-app-statistics