You are on pfsense?
0 bufferbloat really isn't a goal, you need some "Good queue":
https://queue.acm.org/detail.cfm?id=2209336
secondly ack traffic on a very asymmetric connection such as yours can be really problematic, which is why cake has an ack-filter:
https://blog.cerowrt.org/post/ack_filtering/
thirdly, cable has framing issues, not so bad as DSL, but it's one reason why it's very hard to get good rate close to the actually subscribed rate. cake has a "docsis" mode which lets you (on egress) get to within a small fraction (.01%) of the cablemodem rate. Otherwise we recommend about 95% below the configured rate on the up, 85% on the down - and then to get to bed!
Lastly the test that (after we've settled on nearly good values with tcp_nup and tcp_ndown) we aim for good results on is the rrul test, which is a bittorrent-like but even nastier test that tests both up and down and various forms of ping at the same time. You might have thought you've tuned it up properly, but after running the rrul you will see some sideffects both of framing and ack traffic and have to bump both down and up down.
I am very interested in your "before" and "after". Also most likely, all your down bufferbloat is now on your wifi, and I have no idea if ruckis is shipping fq_codel or not. Get 50+ feet from the AP, through a couple walls, and repeat your tests from a laptop, but not outdoors in the middle of the night.
To try and comment on the gigbit down cable results, the responsiveness test result circa 1000 pretty much proves there isn't pie or any form of smart queue management there, and if you are inspired by rainmaker's efforts, slam an opewrt/dd-wrt/etc x86 box in front of it, and enable cake. A tcpdump packet capture can be more revealing.
The fiber test... not sure... 4000 is a pretty good result, partially due to the lower baseline RTT of fiber, partially due to having smaller buffers by default than most cable modems, again a tcpdump and rtt plot from this test would show what the underlying latency under working conditions was. It is possible to do much better than 4000.
Whenever you are next at a friend's house, or a shared wifi space like a cybercafe, try this test there. When I first got out of the house this year and went to a busy coffee shop (not starbucks which at least in these parts has google wifi with all these fixes in it), I was seeing rpm values of 25-45. That's not thousands, or hundreds.... I gave that coffee shop one of my routers in exchange for a few pastries.
The idle bloat is usually an artifact of powersave. Not much you can do about that, and do you care what your car does when it's parked?
Just had a read and there are links to profiles for iDevices too. I installed a profile on my iPad and get a result of 994 on there.
I'd like (someone else!) to produce an open source version of that client with some more features like attempting to isolate where the bloat is coming from - the wifi, the isp, or deeper in the network. Not everything wrong with the network is the ISP's fault. The current version of the apple network quality specification is here:
https://github.com/network-quality/.../master/draft-cpaasch-ippm-responsiveness.txt
And apple's server side code is here:
https://github.com/network-quality/server which will let more people test their backend services. There's a ton of bloat in the cloud, also. A lot of users of microservices have thus far totally missed the benefits of sch_fq in linux, BBR, and for that matter, something as basic as using TCP_NOTSENT_LOWAT.
I note that I did get banned for spam earlier, and I am now, kind of doing SEO on these new efforts by apple, as they have only been released for a week or two, and ultimately we have a whole internet to fix. I hope the moderators don't mind.
Still tweaking.
(Edit: That's because of setting up SQM with cake, not because OpenBSD is rubbish at routing, to be clear!).
