Virgin Media Discussion Thread

Associate
Joined
6 Nov 2021
Posts
43
Location
Bristol
£62


hoping to replace it with edgerouter er6,or something similar any recommnedations welcome.

Zia

er6 won't get you there either at gbit. Also the udm pro's only good to about 700mbit presently (no cake, either)

at 1gbit...

The i7 rainmaker is using is overkill. An i5 or even a modern "celeron" is probably good enough. The APU2 isn't. If you don't want to roll your own, the evenroute pro is due out soon: https://evenroute.com/iqrouter-pro

raspi 4s are pretty good, but I'm not a user of those, nor do I know if they can do a gbit.

Also over here we've been trying to get a hardware offloaded fq_codel implementation to sing and dance: https://forum.openwrt.org/t/validating-nss-fq-codels-correctness/111123/ - the NSS chipset is widely used in a multiplicity of products.

The final world domination event will come from a competent hardware offload of cake, and cost reductions below 80 dollars, which would be, oh, 3-5 more years in the future.

And before that happens all the bloat shifts to the wifi.

IMHO: For the vast majority of personal usage, very few need more than 100Mbit for a typical family of four, including a gamer, and a case could be made for 40mbit.
 
Soldato
Joined
18 Aug 2007
Posts
9,710
Location
Liverpool
The i7 rainmaker is using is overkill. An i5 or even a modern "celeron" is probably good enough.

All amazing info as usual, @Dave Taht and I'm working my way through all the links you post. Just because I can't always keep up with quoting and answering each one, please don't assume they're going unheeded. My usual 'router' box actually runs a Celeron G4560 (Kaby Lake, 2c4t @ 3.5GHz), and it never breaks a sweat (load average 0.0.0). As I briefly alluded above, this i7 box is actually earmarked as a server (LXC/D probably), but I didn't want to hose my original OpenBSD router install before thoroughly testing OpenWrt; not just for its suitability but for my ability to get along with it. As it happens, I needn't have worried so I'll be moving the config over to my original box when I get a chance and repurposing the i7 box appropriately.

I'll drop you those before and after test results as soon as possible, but (not sure I mentioned this on PM?) we have four kids, three of whom are young (2,5,6) so it's very a much an all hands on deck situation at home. I take my free time where and when I can. I certainly haven't forgotten! The flent graphs (thanks again for the use of the London server!) are very instructive.

Cheers.

Edit: BTW, the Raspberry Pi 4 will do gigabit (even with NAT, SQM, etc).
 
Last edited:

TNA

TNA

Caporegime
Joined
13 Mar 2008
Posts
27,577
Location
Greater London
er6 won't get you there either at gbit. Also the udm pro's only good to about 700mbit presently (no cake, either)

at 1gbit...

The i7 rainmaker is using is overkill. An i5 or even a modern "celeron" is probably good enough. The APU2 isn't. If you don't want to roll your own, the evenroute pro is due out soon: https://evenroute.com/iqrouter-pro

raspi 4s are pretty good, but I'm not a user of those, nor do I know if they can do a gbit.

Also over here we've been trying to get a hardware offloaded fq_codel implementation to sing and dance: https://forum.openwrt.org/t/validating-nss-fq-codels-correctness/111123/ - the NSS chipset is widely used in a multiplicity of products.

The final world domination event will come from a competent hardware offload of cake, and cost reductions below 80 dollars, which would be, oh, 3-5 more years in the future.

And before that happens all the bloat shifts to the wifi.

IMHO: For the vast majority of personal usage, very few need more than 100Mbit for a typical family of four, including a gamer, and a case could be made for 40mbit.
Excellent info. I am on a 216Mbit line and can’t see myself needing more than 500 for a very very long time. IQrouter Pro looks a bit overkill for my needs and not exactly cheap. The IQrouter V3 on the other hand seems somewhat dated as it does not offer ax or WPA 3.

Can you recommend something under £200? What would you buy and does stretching a bit more with budget get you something much better? Cheers :)
 
Soldato
Joined
29 Dec 2002
Posts
7,260
Excellent info. I am on a 216Mbit line and can’t see myself needing more than 500 for a very very long time. IQrouter Pro looks a bit overkill for my needs and not exactly cheap. The IQrouter V3 on the other hand seems somewhat dated as it does not offer ax or WPA 3.

Can you recommend something under £200? What would you buy and does stretching a bit more with budget get you something much better? Cheers :)

Consider that you're paying someone to buy a known Chinese OEM's hardware, install a tweaked WRT image and put a sticker on it and support it/test builds going forward - nothing at all wrong with that and it certainly has a value while offering a near plug and play option/reduced maintenance, but if you know enough to care, you probably care enough to learn/tweak DIY? You can buy the same/better hardware and do much better with a dedicated AP installed in a central location, heck you could do all that for way less money up front. You can get some very capable ex-corp. desktop's for very little if you time it right, I was sent a BIN offer on a i5 8400 (6 core) Vostro for £150 last night and the i3 8100 (4 core) I was watching last week went for £100ish. Massive overkill for a router, but not that much more expensive than something 'cheap' and make ideal personal ESXi/Proxmox hosts for R210-II money. The only down side is a slightly higher power bill, but vastly more capable.

I'm off to throw a new router build together and begin playing, I haven't given DD/O WRT on x86/x64 much attention previously, the last time I bothered with anything WRT was ARM back in 2014? ish, I may have been guilty of complacency and a little tunnel vision in my quest to make pf/OPN more UTM like and ultimately settling on Untangle for simplicity. From what i've just read VM are sending me a SH4, so no need to consider 2.5Gb/NBASE-T till they move Gig2 out of trials :(
 

TNA

TNA

Caporegime
Joined
13 Mar 2008
Posts
27,577
Location
Greater London
Consider that you're paying someone to buy a known Chinese OEM's hardware, install a tweaked WRT image and put a sticker on it and support it/test builds going forward - nothing at all wrong with that and it certainly has a value while offering a near plug and play option/reduced maintenance, but if you know enough to care, you probably care enough to learn/tweak DIY? You can buy the same/better hardware and do much better with a dedicated AP installed in a central location, heck you could do all that for way less money up front. You can get some very capable ex-corp. desktop's for very little if you time it right, I was sent a BIN offer on a i5 8400 (6 core) Vostro for £150 last night and the i3 8100 (4 core) I was watching last week went for £100ish. Massive overkill for a router, but not that much more expensive than something 'cheap' and make ideal personal ESXi/Proxmox hosts for R210-II money. The only down side is a slightly higher power bill, but vastly more capable.

I'm off to throw a new router build together and begin playing, I haven't given DD/O WRT on x86/x64 much attention previously, the last time I bothered with anything WRT was ARM back in 2014? ish, I may have been guilty of complacency and a little tunnel vision in my quest to make pf/OPN more UTM like and ultimately settling on Untangle for simplicity. From what i've just read VM are sending me a SH4, so no need to consider 2.5Gb/NBASE-T till they move Gig2 out of trials :(
As I recall you like to advocate using gear the ISP provides, but those tend to slow and I am not sure what kind of traffic management they offer. Like is the SH4 any good on its own? I know SH3 had rubbish Wi-Fi and hardly any customisable options, just barebones rubbish which would be fine if all I had was wired connections.

I fancied getting Dave’s opinion as he seems to be on another level when it comes to networking knowledge. Always good to get more opinions :)

I get what your saying about building my own router essentially, but as you say that seems like overkill and will consume a lot more time and electricity. Plus if I mess something up by accident I may be worse off. Not interested in 1gig let alone 2gig lines in the foreseeable future. Nice to have sure but money down the drain as far as I am concerned as I do not download much these days and 100meg was bloody fast already but now I have double that I feel spoilt. When 500meg can be had for £25 I will eventually get on that.

When you say “know enough to care” what do you mean exactly? Are you saying ASUS and other third party hardware all have back doors or something? Would no one have picked up on such a thing by now? Wonder what @Dave Taht take on this is in addition to my original question :)

As an example the RT-AC86U with merlin firmware you can use Cake SQM which seems to be very good and even Rainmaker which is super into this stuff did not know about that from what I can tell.
 
Soldato
Joined
18 Aug 2007
Posts
9,710
Location
Liverpool
As an example the RT-AC86U with merlin firmware you can use Cake SQM which seems to be very good and even Rainmaker which is super into this stuff did not know about that from what I can tell.

I knew about it mate I just couldn't use it on OpenBSD. Having switched back to Linux (OpenWrt) though, I'm enjoying it very much. Here's my progress so far. The little ping spike at the start coincides with the usual burst of (up or down) speed VM seem to apply at the start of each stream. On the unscaled graph it shows it actually only hit late 20s milliseconds, so nothing ostentatious. I *can* smooth it out but at the cost of overall rate stability and throughput, so I left it as it is. Not a bad graph as far as I can tell (but I can't tell very much). :p

ZobjpoB.png

Edit: And without any SQM applied, just for completeness.Notice how the latency jumps from ~17ms at idle and into the 40s under load. That doesn't happen with SQM enabled (above), rather only a few ms get added and it's consistent.

poxjwW1.png
 
Last edited:
Soldato
Joined
29 Dec 2002
Posts
7,260
As I recall you like to advocate using gear the ISP provides, but those tend to slow and I am not sure what kind of traffic management they offer. Like is the SH4 any good on its own? I know SH3 had rubbish Wi-Fi and hardly any customisable options, just barebones rubbish which would be fine if all I had was wired connections.

I think you may have missed the context of my previous posts, I advocate people buying hardware that suits what they want to do, quite often that's not the same as what they think they need to buy. Usually that tends to look like 'I need a new router' followed by asking them what specifically they want a new router to do that the existing one doesn't, that's when it usually becomes clear it's nothing to do with the router side of things and everything to do with wifi coverage/range/speed. They've usually not done a survey, don't know about the noisy neighbours on an overlapping channel, have the router in a poor location and the client will generally be some sort of streaming device sandwiched between a solid wall and an RF shielded TV or that the 99p USB N spec dongle they've put in the back of that nice metal PC case and only sticks out 7mm contains the wireless chipset and antennae they are connecting to the network with and they aren't sure why it's not giving them gigabit. In situations like that, replacing a router broadcasting at the maximum permitted power level with another one doing the same in the same location with the same power level is probably not ideal. Instead spending the money on a decent AP installed in a central location with a wired backhaul will usually be transformative by comparison, especially if happens to handle slower clients hogging the radio's more elegantly. That's not the same as advocating using 'gear the ISP supplied', it's responsible recommendations for the actual issue. The answer would change as the usage case/budget did.

The SH4 can do over gigabit with a suitable wifi client/environment (very much to my amazement when I saw the picture), then again an online speed tests is not exactly a reliable reference point, let alone comparable to a full iperf session. For all people love to moan about the SH3 wifi, it's not been that bad for me. Yes I run Ubiquiti AP's, but that's because it's unreasonable to expect a hub sat in my downstairs lounge to cover another two floors above it. In terms of options, it's a basic router designed to provide a basic service, if you've ever done any form of TS, the simpler the better for the majority. If you want non basic, you put it in modem mode and spec your own kit accordingly. Yes the underlying hardware issues Intel inherited from the TI purchase are still a thing, but the mitigation means in general use they are not noticeable.

I fancied getting Dave’s opinion as he seems to be on another level when it comes to networking knowledge. Always good to get more opinions :)

You're 100% right, he absolutely is. But this isn't a software/networking question, it's a hardware question. He's also on a different continent with vastly different new/used hardware market, so asking him for advice on hardware with a GBP budget is like me asking you for hardware sourcing advice in USD. A CPU is still a CPU, but domestic pricing and distribution differ drastically. I can't stress how different the US is in terms of hardware, I used to joke in the UK I have a room with half a rack, in the US i'd likely have half an isle in a DC as the abundance and price of decent hardware relative to what we pay over here can be eye watering, a bit like the power bills to run it in some states... and don't even get me started on the 120v.

Your first suggestions on hardware candidates didn't demonstrate the greatest of awareness of suitable hardware or it's capabilities/features and you backed away from the price of an off the shelf solution, I took it that pointing out you could buy and install WRT yourself on equally good if not better hardware for way under budget was the advice you needed, you didn't seem to have grasped that you were looking at a generic Quotom/Procelli PC with stickers (mmm stickers) and a custom WRT install. But the thing that I want to stress here is for 200mbit, hardware is not the issue you seem to think it is, beyond a non ancient x64 CPU and a modest amount of RAM/not awful NIC, almost anything from roughly the last decade should comfortably do 200Mbit with QoS.

I get what your saying about building my own router essentially, but as you say that seems like overkill and will consume a lot more time and electricity. Plus if I mess something up by accident I may be worse off. Not interested in 1gig let alone 2gig lines in the foreseeable future. Nice to have sure but money down the drain as far as I am concerned as I do not download much these days and 100meg was bloody fast already but now I have double that I feel spoilt. When 500meg can be had for £25 I will eventually get on that.

That may be what you chose to read, but it's really not what I said - the devil is in the detail.

It took me longer to walk down two floors to get a screwdriver and come back up than it did to 'build' a router today. Physical build was under 2 minutes. I removed two screws, slid the lid off a (£59.99 delivered) Lenovo S510 (i3-6100/4GB/500GB HDD/DVDRW), lifted the retaining bracket over the PCIe blanking plate, slotted an i350-T4 v2 in (because I haven't worked out where the T2 is hiding and wasn't playing hide and seek in the rack), closed the retaining bracket, unplugged the power to the DVDRW/HDD and put the lid back on and replaced the two screws. While I was doing that, the USB image was written and verified. Power wise the little Quotom boxes are generally round the 10-18w mark depending on spec/load, i'm always wary of the low end passive cooled versions, they have an unfortunate history and 24/7/365.25 can be problematic if they are under load constantly. My S510 was around 25w? idle last time I looked (ESXi with HDD), I wouldn't imagine it'll drop massively without DVD/HDD, but for the sake of argument even if we say 10w for a Quotom and 20w for the S510 I still have over a decade of idle running - at todays power prices (ex VAT) - before i'm financially worse off and I have £140.01 in my pocket to dry my tears with.

When you say “know enough to care” what do you mean exactly? Are you saying ASUS and other third party hardware all have back doors or something? Would no one have picked up on such a thing by now? Wonder what @Dave Taht take on this is in addition to my original question :)

As an example the RT-AC86U with merlin firmware you can use Cake SQM which seems to be very good and even Rainmaker which is super into this stuff did not know about that from what I can tell.

I didn't mention ASUS :confused: My comment was on the basis that if you know enough to care about running a highly configurable dedicated x64 based router, you probably care enough to learn how to set-up and tune/tweak it for your needs, as opposed to buying a pre-installed device.

Every single vendor has had or will have issues and be impacted by a CVE eventually, most respond promptly and move on, that's how it's supposed to work. As far as ASUS go, it's well documented they have a horrible history when it comes to networking products, they ignored known issues for years, left users with data exposed to the internet, faked FCC test data (caught and fined) and have had issues since the beginning with interfaces dropping off routers (still a thing after 6+? generations), along with years of failing to address the Mediatek modem chipset issues (over two generations) while everyone else who used them managed to get them working and only bothered to try and fix the security issues when a reseller threatened to remove them from the platform and the US Gov. began investigating them. Ultimately they had to agree to 25 years of external auditing/fines. That's without the joke that is the RMA side. Even Merlin advises against opening up things like external admin on the ASUS (and by implication his) fork in his release notes. How many times does the same company have to do really bad things and keep making the same hardware/software mistakes and being terrible to deal with before you consider them unworthy of your money and more importantly time? I use certain companies because they're good at dealing with issues if/when they occur, if that costs me more more money, so be it. I can always make more money, I can't make more time.
 
Soldato
Joined
18 Aug 2007
Posts
9,710
Location
Liverpool
You took a major upload bandwidth hit for some reason.

Your baseline latency under load is quite good. I wonder what it's like on their lower tiers of service.

Yeah it goes from around 10Mbps per stream to 8.5 or so. I don't see much impact in the real world, as far as I can tell. Here's ThinkBroadband's speed tester, showing the speed of 1x stream, multi-streams and upload, with latency during testing overlaid:

WTmzWMP.png
Sbf01L6.png
szdX7Rz.png

Here's a speedtest.net without SQM (base line rate):
LnozdFC.png

Here's one with SQM applied:
yoMpIIS.png

I tend to find more variability due to time than anything else. At the moment, writing this, I'm seeing A+ bufferbloat on DSLReports speed test even with no SQM. Most other times it's a D. One thing I do notice, is that the Waveform bufferbloat test *always* shows a couple of high pings on the latency bar (up and downstream) regardless of settings. For example 99% of the dots are in the 17ms to 23ms range but there'll be one or two in every test sat outlying around 80ms (sometimes even 400ms!). I haven't been able to eliminate those, regardless of SQM speeds, and my rating on Waveform varies between A with 'may have issues due to bloat in online gaming' (all other categories get a tick/check), and a full house of checks with an A rating and no issues reported. That variability again, as per DSLReports, regardless of settings.

I'm not sure I got the link layer adaptation settings right by the way, as the documentation contradicts itself a little in places. I set it to Ethernet with overhead, and then per packet overhead to 18 (cable), and that was it. One OpenWrt docs page suggested changing MPU to 64 also, but I found it made little (or even negative) difference. Another page (SQM details, iirc) says that with later OpenWrt versions it's redundant. So now I just have the packet size set and advanced options left unchecked. Is that correct?
 
Associate
Joined
6 Nov 2021
Posts
43
Location
Bristol
do you have ack-filter on, on the upload side?

The speedtest result has the always misleading test of idle latency. And most browsers really struggle at a gbit.

as for the docsis keyword, I don't know. Usually it's helpful.

Anyway I'm hoping at this point you and your family can feel a difference in overall performance, you can torrent and game to hearts content, and then move on towards debloating Britain. :)
 
Associate
Joined
6 Nov 2021
Posts
43
Location
Bristol
Also, it sounds like you've already trained yourself past this, but the default from most ISPs in a blind urge for the best bandwidth core on a speedtest, was basically
giving you a car that could run at 200kph, but not steer, or brake, or let you change the station on the radio, at that speed. By giving yourself a bit of headroom at 180kph, and fair queuing and aqming the link, you restore your ability to do all that.
 
Soldato
Joined
18 Aug 2007
Posts
9,710
Location
Liverpool
do you have ack-filter on, on the upload side?

The speedtest result has the always misleading test of idle latency. And most browsers really struggle at a gbit.

as for the docsis keyword, I don't know. Usually it's helpful.

Anyway I'm hoping at this point you and your family can feel a difference in overall performance, you can torrent and game to hearts content, and then move on towards debloating Britain. :)

Also, it sounds like you've already trained yourself past this, but the default from most ISPs in a blind urge for the best bandwidth core on a speedtest, was basically
giving you a car that could run at 200kph, but not steer, or brake, or let you change the station on the radio, at that speed. By giving yourself a bit of headroom at 180kph, and fair queuing and aqming the link, you restore your ability to do all that.

Haha yes you're right. The shock of seeing speed tests below 900Mbps was uncomfortable at first, as I've spent so long pushing to get the absolute maximum throughput, all else be damned. Now I'm perfectly happy seeing 750Mbps if it means latency, jitter and bloat are optimal. No point being fast if your throughput bounces around and one person using the line kills it for everyone else. ;)

As it happens, I was only using luci. After a quick rtfm, and reading a post you made about SQM on cable over at Reddit, I've disabled link adaptation (per your advice on Reddit), added some parameters to 'advanced option string' on egress and ingress, and now that 'docsis' is added to the switches I've increased the SQM target to line rate... All of which translates in /etc/config/sqm to:

Code:
config queue 'eth1'
        option qdisc 'cake'
        option script 'piece_of_cake.qos'
        option verbosity '5'
        option interface 'eth0'
        option download '950000'
        option upload '52000'
        option enabled '1'
        option ingress_ecn 'ECN'
        option qdisc_advanced '1'
        option squash_ingress '1'
        option iqdisc_opts 'docsis besteffort ingress nat noatm'
        option qdisc_really_really_advanced '1'
        option eqdisc_opts 'docsis ack-filter nat noatm'
        option egress_ecn 'NOECN'
        option squash_dscp '1'
        option debug_logging '1'
        option linklayer 'none'

Before, I didn't have the docsis or ack-filter lines in there. Now, page loads seem not just snappy but instant - even on my 2012 Core i5 MacBook Pro, which normally struggles in macOS (but not Linux!). I haven't slept in two days properly (not just testing, ha!) so I'll be dusting off flent et al. tomorrow to see how it affected things objectively.

I have a feeling the change will be positive. A quick and dirty extended speed test shows increased downstream and basically line rate upstream (49-50Mbps), meanwhile ping running in terminal alongside shows almost no variation between unloaded and loaded (14ms vs 18ms) now. Adding noatm got me some extra downstream and 1Mbps extra upstream at seemingly no other penalty.

Thanks so much again for taking time to share your expertise and help me (us) out. I promise to keep on passing it on.
 
Last edited:
Associate
Joined
6 Nov 2021
Posts
43
Location
Bristol
Win!!!! If you want DSCP markings to work for you on egress through the router:

option squash_dscp '0'

If you want to try ecn on your client, see https://www.bufferbloat.net/projects/cerowrt/wiki/Enable_ECN/ and on most OSX implementations in addition
to the ones documented there:

sudo sysctl -w net.inet.tcp.disable_tcp_heuristics=1

Nobody knows if ecn will deploy widely, but you will see slightly less jitter on your tcp flows. Probably. I would do the ecn test before fiddling with the dscp.
 
Soldato
Joined
18 Aug 2007
Posts
9,710
Location
Liverpool
Win!!!! If you want DSCP markings to work for you on egress through the router:

option squash_dscp '0'

If you want to try ecn on your client, see https://www.bufferbloat.net/projects/cerowrt/wiki/Enable_ECN/ and on most OSX implementations in addition
to the ones documented there:

sudo sysctl -w net.inet.tcp.disable_tcp_heuristics=1

Nobody knows if ecn will deploy widely, but you will see slightly less jitter on your tcp flows. Probably. I would do the ecn test before fiddling with the dscp.

Thanks again (again!), Dave. :) One thing I have noticed, is that if I run (eg) ping 1.1.1.1 in a terminal (I'm on Linux, primarily) and seed Ubuntu torrent to test, the pings start to swing much more than flent's graph suggests. I've snipped out a lot of the irrelevant ones (jumping back to the ~20ms baseline as it always does), and left in some of the more interesting swings. The seq numbers are unaltered so you can get an idea where in the list the swings happen:

Code:
ping 1.0.0.1
PING 1.0.0.1 (1.0.0.1) 56(84) bytes of data.
64 bytes from 1.0.0.1: icmp_seq=24 ttl=60 time=19.1 ms
64 bytes from 1.0.0.1: icmp_seq=25 ttl=60 time=21.9 ms
64 bytes from 1.0.0.1: icmp_seq=26 ttl=60 time=18.5 ms
64 bytes from 1.0.0.1: icmp_seq=27 ttl=60 time=19.2 ms
64 bytes from 1.0.0.1: icmp_seq=28 ttl=60 time=36.5 ms
64 bytes from 1.0.0.1: icmp_seq=29 ttl=60 time=19.3 ms
64 bytes from 1.0.0.1: icmp_seq=30 ttl=60 time=46.5 ms
64 bytes from 1.0.0.1: icmp_seq=31 ttl=60 time=20.5 ms
64 bytes from 1.0.0.1: icmp_seq=32 ttl=60 time=42.9 ms
64 bytes from 1.0.0.1: icmp_seq=41 ttl=60 time=51.9 ms
64 bytes from 1.0.0.1: icmp_seq=42 ttl=60 time=28.2 ms
64 bytes from 1.0.0.1: icmp_seq=43 ttl=60 time=55.6 ms
64 bytes from 1.0.0.1: icmp_seq=44 ttl=60 time=23.8 ms
64 bytes from 1.0.0.1: icmp_seq=45 ttl=60 time=69.3 ms
64 bytes from 1.0.0.1: icmp_seq=46 ttl=60 time=44.5 ms
64 bytes from 1.0.0.1: icmp_seq=47 ttl=60 time=20.8 ms
64 bytes from 1.0.0.1: icmp_seq=48 ttl=60 time=29.9 ms
64 bytes from 1.0.0.1: icmp_seq=49 ttl=60 time=19.6 ms
64 bytes from 1.0.0.1: icmp_seq=50 ttl=60 time=21.1 ms
64 bytes from 1.0.0.1: icmp_seq=51 ttl=60 time=23.9 ms
64 bytes from 1.0.0.1: icmp_seq=52 ttl=60 time=19.0 ms
64 bytes from 1.0.0.1: icmp_seq=65 ttl=60 time=132 ms

I mean these are hardly wild and massive swings, and they always jump right back to ~20ms right after, as though they're being 'reined in', for want of a better way to explain it. Is this what I should expect to see (and I can stop spending time trying to 'fix' it), or is this a sign my rates are set too high in SQM? Once I added docsis to the parameters in /etc/config/sqm I basically just whacked in line rate (950000 and 50000) and that gave the previously shown excellent flent graph. Browsing is instant/snappy, and everyone's able to stream HD video (social media, Invidious etc) instantly despite torrents running. So it 'seems' OK, but I got the impression ping should stop fluctuating like that? Everything else looks good, so I wonder am I worried (hyperfocusing) about nothing.

If I drop the rates down again (eg 890000 or even 820000) and drop upstream commensurately, the flent graphs never look so good as they do with it set almost at line rate. Again, sorry for the hassles and questions - it's just fun to learn and play with. I bet you're sorry you came here now, ha! :D

Edit: Sorry I just realised, I did both tests (seeding an Ubuntu torrent and pinging 1.0.0.1) on the same machine... Which has WireGuard VPN enabled to Mullvad (i.e. UDP, no qdisc in the OS). When I SSH'd to my Alma/RHEL server across the LAN and ran ping from there, with the torrent running on my main desktop, pings look great (18-20ms mostly). I think I've mixed up a 'no qdisc and lots of udp traffic flooding at line rate' with 'possible SQM issue' here. The other machines (without VPN) show no such issues.
 
Associate
Joined
6 Nov 2021
Posts
43
Location
Bristol
I wouldn't use the term "flooding" as torrent's underlying transport has the LEDBAT congestion controller. LOTS of UDP encapsulated entering into a single AQM, perhaps.

I've said elsewhere we'd made site-2-site wireguard work pretty well with fq-codel, but was unclear about saying if that was actually the bottleneck or not.

The to-a-client vpn case is actually kind of important. In that case you essentially only have one FQ lane at the router for your entire desktop, and the AQM portion of cobalt (cake's reimplementation of the codel AQM) is supposed to get things under control there, eventually. In your case it didn't. Now the bloat in this case could be happening in two places, at the router (which only sees a single queue), or within the wireguard instance (was this userspace or the kernel wireguard?), and in either case there's an inflated baseline RTT to begin with, so congestion control operates more slowly. (I imagine your vpn's termination point is 10-20ms away?)

It really shouldn't take as long as you describe for cobalt at the router to have got control of that queue, but I won't rule it out. More likely, however:

In the kernel implementation of wireguard, jason optimized for cores, not queues, and there is a point at which packets accrue within wireguard (when you cannot encrypt as fast as packets can come in), that's hard to hit, but once you hit it that's where bloat happens. The testing they did was also dominated by one way tests (sigh), not a mixture of large and small packets you typically see over torrent, or the rrul tests. We'd proposed making wireguard fair queued at least, internally, at several points in its development. The plot here shows the 'compute bloat',
when I tested it, running directly on a low powered router, in the early days. https://blog.cerowrt.org/post/wireguard/

If you want to try running a rrul test through it, that ought to be "interesting".

So far as I know the userspace implementation's users are seeing signs of bufferbloat within wireguard. As it's potentially far easier to do fq in userspace, I'd been
considering tackling it in the go or rust versions at some point to see if I could demonstrate a benefit.

Anyway, the rest of your network is fine.
 
Last edited:
Associate
Joined
6 Nov 2021
Posts
43
Location
Bristol
I have been trying to escape my bubble and understand real problems real people have to direct my future research. So I'm glad I came, though i'd rather like to pop over to the Fleece for a cold one and a live band this evening, y'all are 6? 7000 miles away..

"If I drop the rates down again (eg 890000 or even 820000) and drop upstream commensurately, the flent graphs never look so good as they do with it set almost at line rate. Again, sorry for the hassles and questions - it's just fun to learn and play with. I bet you're sorry you came here now, ha!"

Getting very close to line rate is important for cable especially. We are plunking a sophisticated deficit based shaper in front of a complicated request/grant based shaper that has to do a *predictive* request/grant access to the medium for X amount of bytes back to the CMTS. Fooling it into doing the right things, makes for really nice things to happen, but in the case where your wire gets oversubscribed it's going to break down without some form of passive measurement of the real rate, or (preferably) a control message from the isp saying we need to lower your rate to cope with our load, or adoption of cake rather than pie, in the cablemodem itself.

The request/grant pattern is why I don't expect to ever get less than the approximately 2.5ms jitter you are observing. DOCSIS 4.0-LL can reduce that jitter further, but thus far it seems to be pitched at "low latency as a service", rather than something everyone will get. I kind of expect that once pie rolls out it will become difficult to control things this well from your own firewall.

Starlink, wifi, lte all share a request/grant pattern, as do some forms of fiber. Fiber at least is full duplex, and we'll have full duplex on these other technologies "real soon now".
 
Back
Top Bottom