Weird packet loss issue

digitalwolf

digitalwolf

digitalwolf

Soldato
Joined
27 Mar 2004
Posts
3,601
Location
England
I have been running a Flint 2 MT6000 on my
Virgin Gig1 connection since they were released. It has been rock solid…..until now.

I flashed vanilla Openwrt on it last year and decided last week to upgrade from 24.10.6 to 25.12.4 and ever since I have been plagued with a weird packet loss problem which has left the connection unstable.

I have ran ping plotter, directly wired into the 2.5gb LAN1 port on the Flint and the packet loss appears to be coming from HOP 2, which I believe is my local Virgin CMTS. It’s sporadic and happens during the night too. It is causing buffering, making work from home difficult but without the packet loss, the connection is performing well still pulling over 1Gbps and the latency is good and the jitter is stable.

I don’t really like running the SH5 in router mode as my Openwrt configuration is highly customised with multiple VLANS etc but I put it in router mode this afternoon and the connection is rock solid. No packet loss at all. BQM looks stable.

Back into modem mode with the Flint, it’s dropping packets again. Red packet loss spikes and red dots across the top of the BQM.

I have tried rolling back to 24.10.6 with my config backed up. Tried a brand new fresh build. Tried GL-iNets firmware, 4.8.4 and the 4.9.0 beta. They’re all the same. No difference.

I have checked the logs on the Flint and it’s not giving me any clues at all. No disruption on the WAN port. It doesn’t appear to be flapping. I have tested my cat6 with a cable tester, it has passed.

I am stumped and apart from buying a replacement router, I’m not sure what to check next.

It could be the WAN port on the SH5 but the WAN port on the Flint 2 does not appear to be dropping.

Is there anything else I can check on the Flint 2 to see if the SH5 wan port is flapping or the disruption is being caused by the cable modem?
 
John24 said:
When you tried a brand new fresh build was that stock defaults with minimal changes to get the connection working?
Click to expand...
Yes basically WAN configuration, set up 2.4Ghz and 5Ghz and left it.

I ordered an ASUS TUF to test and it’s not an issue with the super hub 5 it seems. The ASUS has not dropped any packets.

Plug the Flint in, it starts dropping packets again.

I flashed the new GL-iNet OpenWrt24 firmware to the router and it was stable until I plugged my LAN devices in. I’m going to try leaving them disconnected and just the WAN Ethernet plugged in from the VM side.

Another suggestion I have seen is changing the second 2.5gbe to a WAN port and trying that.

Frustratingly the router has just gone out of warranty so I hope it’s not a hardware issue.
 
Last edited:
Checked the logs today and noticed when the WAN connection drops briefly and I get packet loss, it coincides with….

Maximum number of concurrent DNS queries reached (max: 150).​


Now it could be rogue device hammering the DNS or it could be the WAN dropping.

Nothing seems to be dropping on the cable modem side according to the logs, the power levels and SNR all seem fine and the Flint is not reporting the WAN interface is dropping.

Bit stuck with this one.
 
Last edited:
Test connectivity using IP addresses preferably from machine cabled directly into the Flint 2 router. Ping test to an internet address, VM router, and Flint router e.g.:

ping -t 8.8.8.8 (google DNS)
ping -t 192.168.0.1 (if VM router in router mode, or 192.168.100.1 if modem mode, note when internet connection is down will be unable to get to VM router management address 192.168.100.1 if using modem mode)
ping -t 192.168.1.1 ( or Flint 2 LAN IP address)

Leave the ping tests runnning and when you expecirence an issue it will help highlight if internet connection was dropped or both LAN and WAN etc.


Things to check on Flint 2 when issue happens:

- check system uptime (in case the router lost power unexpectedly)
- check CPU usage (if it's maxing out it can cause packet loss)
- check memory usage (if low or running out of available memory)
- check active connections (scroll down on Overview page to active connections or go to Status -> Realtime graphs -> Connections), check not excessive number of connections reached or if anything looks unusual.
- check bandwidth usage, go to Status -> Realtime graphs -> Bandwith ( check if any links are getting saturated / maxed out can cause packet loss issues on that port e.g. if you are doing inter-vlan routing and a backup is running and maxing out the physical ports etc.)
- you can also SSH onto the Flint 2 router and do similar ping tests to devices on LAN and WAN to see if also getting packet loss, e.g. if you have a managed switch or Access Point cabled in you can ping their management IP addresses usually a good test.

VM connection:

- if you are maxing out your bandwidth can also cause packet loss / intermittent or inconsistent response times / lag / delays etc. This I think is more noticeable when the upstream bandwith is getting maxed out e.g. if a deivce is uploading to the cloud.

Further checks:

- you can install tcpdump on the Flint 2 can give you more clues of network traffic what is going on. e.g.: some examples:
"tcpdump -ni eth1 icmp and host 8.8.8.8" (eth1 is WAN port)
"tcpdump -ni br-lan icmp" (check ping packets icmp, can filter traffic for specific host IP etc.. )
"tcpdump -ni br-lan port 53" (port 53 is for DNS traffic)


If you have an internal DNS server like raspberry pi hole, check if that is having any issues e.g. cpu usage/memory, do similar network connectivty tests etc, because slow DNS server can also have a big impact on user experience.

The IP tests should help narrow down if connectivity related. You can test DNS related using nslookup to different servers (e.g. 8.8.8.8 or 8.8.4.4) and see reponse times etc to DNS queries or use external DNS servers instead of an internal DNS server if you are using one.

In the past I experienced an old android device (Nexus 9) would spam DNS requests (about 3K per second) when there was no interenet connection and it was causing the CPU on the router to max out, i used tcpdump it helped highlight what was going on the network.

Im on Virgin Media connection and have been experiencing internet connection drops the past week but i suspect was related to the recent heat wave as its been working ok since. You could also log into your VM online account see if there are any known issues reported in your area.
 
Last edited:
os4321 said:
Test connectivity using IP addresses preferably from machine cabled directly into the Flint 2 router. Ping test to an internet address, VM router, and Flint router e.g.:

ping -t 8.8.8.8 (google DNS)
ping -t 192.168.0.1 (if VM router in router mode, or 192.168.100.1 if modem mode, note when internet connection is down will be unable to get to VM router management address 192.168.100.1 if using modem mode)
ping -t 192.168.1.1 ( or Flint 2 LAN IP address)

Leave the ping tests runnning and when you expecirence an issue it will help highlight if internet connection was dropped or both LAN and WAN etc.


Things to check on Flint 2 when issue happens:

- check system uptime (in case the router lost power unexpectedly)
- check CPU usage (if it's maxing out it can cause packet loss)
- check memory usage (if low or running out of available memory)
- check active connections (scroll down on Overview page to active connections or go to Status -> Realtime graphs -> Connections), check not excessive number of connections reached or if anything looks unusual.
- check bandwidth usage, go to Status -> Realtime graphs -> Bandwith ( check if any links are getting saturated / maxed out can cause packet loss issues on that port e.g. if you are doing inter-vlan routing and a backup is running and maxing out the physical ports etc.)
- you can also SSH onto the Flint 2 router and do similar ping tests to devices on LAN and WAN to see if also getting packet loss, e.g. if you have a managed switch or Access Point cabled in you can ping their management IP addresses usually a good test.

VM connection:

- if you are maxing out your bandwidth can also cause packet loss / intermittent or inconsistent response times / lag / delays etc. This I think is more noticeable when the upstream bandwith is getting maxed out e.g. if a deivce is uploading to the cloud.

Further checks:

- you can install tcpdump on the Flint 2 can give you more clues of network traffic what is going on. e.g.: some examples:
"tcpdump -ni eth1 icmp and host 8.8.8.8" (eth1 is WAN port)
"tcpdump -ni br-lan icmp" (check ping packets icmp, can filter traffic for specific host IP etc.. )
"tcpdump -ni br-lan port 53" (port 53 is for DNS traffic)


If you have an internal DNS server like raspberry pi hole, check if that is having any issues e.g. cpu usage/memory, do similar network connectivty tests etc, because slow DNS server can also have a big impact on user experience.

The IP tests should help narrow down if connectivity related. You can test DNS related using nslookup to different servers (e.g. 8.8.8.8 or 8.8.4.4) and see reponse times etc to DNS queries or use external DNS servers instead of an internal DNS server if you are using one.

In the past I experienced an old android device (Nexus 9) would spam DNS requests (about 3K per second) when there was no interenet connection and it was causing the CPU on the router to max out, i used tcpdump it helped highlight what was going on the network.

Im on Virgin Media connection and have been experiencing internet connection drops the past week but i suspect was related to the recent heat wave as its been working ok since. You could also log into your VM online account see if there are any known issues reported in your area.
Click to expand...
Thanks for your detailed response.

I have done some more digging and have found the following......

A constant ping to 192.168.100.1 does not really highlight anything, apart from sporadic leaps in latency from time to time from 1ms to 27ms for example.

The connection is hardly being touched, it is only really me that hammers it with game downloads etc. The other half just streams ITVX, Netflix and uses the internet for work, teams calls etc.

I am using cloudflare for DNS 1.1.1.1 but have tried Virgin's DNS and Google.

We have a Virgin stream box in the kitchen which was buffering badly 6-7pm yesterday evening. It kept saying your connection is not fast enough. I ran a speedtest during this time and got 1.1Gbps. On my BQM, there was a little packet loss but nothing I would say was major but the stream box didn't like it, that's for sure.

I ran ping plotter to bbc.co.uk for hours last night, directly plugged into the 2.5Gbe port on the Flint and the only packet loss was on Hop2, which I believe is the VM CMTS?

What I am struggling to get my head around is everything was fine until I flashed Openwrt 25.10.2. I did power down the cable Super Hub 5 at the time and since then, I have had the packet loss issue which is sporadic but my actual BQM is significantly different. There are large yellow bars now present throughout the day and that is not device dependent. They show on the SH5, Asus TUF 9400, Flint 2. It's as if the connection has gone 'jittery'.

I have a couple of questions you may be able to clarify for me on the cable side. My knowledge on DOCSIS and how the VM network works isn't great.

Could the SuperHub 5 mask the packet loss and the Flint 2 is just basically replicating the raw connection so to speak? I noticed buffering when using the Super Hub 5 in router mode and thought oh it'll be the wifi. It has made me think it probably wasn't. Nothing showed on my BQM though.

I bought an ASUS TUF 9400 as a test device and noticed I got a new IP with that device. Yet when I plug the Flint in, I get the IP I was assigned back in 2024. I don't get the packet loss on the Asus. I do get the yellow blocks of latency.

Could I be routed differently or be connected to a different server at the VM CMTS depending on what device I use?
 
The packet loss on Hop2 could be a red herring, i also get no response on hop2 to bbc.co.uk, some ISPs configure their network equipment to not respond to ping requests. If there was packet loss at hop2 you would also see packet loss at following hops afterwards and packet loss with ping requests to the remote host (bbc.co.uk).


If the Virgin stream box responds to ping requests, you could SSH onto the Flint 2 router and ping the Virgin stream box check for any packet loss or high response times. If using the Flint 2 for wifi it will show you connected devices Rx and Tx connected wifi Mbps speed, so check speed for the Virgin stream box, if these are low (actual speed will be around 20% less due to wifi overhead sometimes even 50% overhead if congested wifi environment). Could also try a different wireless channel if you suspect wifi issue or move the Virgin stream box to another position/open position if its currently tucked away in a cabinet etc. Also remember to set your country code to GB on the Flint 2 for wifi, i think by default openwrt doesnt set a country code. Also some devices dont like 160Mhz channel width on 5Ghz so could try 80Mhz see if that helps.


I think WAN IP is assigned by MAC address, you could make note of the Asus router WAN MAC address and clone that MAC address on the Flint 2 WAN interface, should get assigned same IP as Asus router i believe. Im not sure how internal VM network infrastructure works.
 
os4321 said:
The packet loss on Hop2 could be a red herring, i also get no response on hop2 to bbc.co.uk, some ISPs configure their network equipment to not respond to ping requests. If there was packet loss at hop2 you would also see packet loss at following hops afterwards and packet loss with ping requests to the remote host (bbc.co.uk).


If the Virgin stream box responds to ping requests, you could SSH onto the Flint 2 router and ping the Virgin stream box check for any packet loss or high response times. If using the Flint 2 for wifi it will show you connected devices Rx and Tx connected wifi Mbps speed, so check speed for the Virgin stream box, if these are low (actual speed will be around 20% less due to wifi overhead sometimes even 50% overhead if congested wifi environment). Could also try a different wireless channel if you suspect wifi issue or move the Virgin stream box to another position/open position if its currently tucked away in a cabinet etc. Also remember to set your country code to GB on the Flint 2 for wifi, i think by default openwrt doesnt set a country code. Also some devices dont like 160Mhz channel width on 5Ghz so could try 80Mhz see if that helps.


I think WAN IP is assigned by MAC address, you could make note of the Asus router WAN MAC address and clone that MAC address on the Flint 2 WAN interface, should get assigned same IP as Asus router i believe. Im not sure how internal VM network infrastructure works.
Click to expand...
On ping plotter and on a normal traceroute hop 2 does respond and it is my Virgin CMTS by the looks. It generally shows packet loss on that one hop though and like you say, not further down the chain.

I added the CMTS’s IP to collectd-mod-ping on Openwrt and it doesn’t respond. It just constantly times out which is weird, as it does respond on a traceroute.

In regards to the WiFi, it’s running on CH100, all my neighbours are down at CH36 so congestion is fine. The stream box is literally 4 meters away from the Flint, albeit in a store cupboard. The signal strength reported on the stream box is 100%. The 1.1Gbps speedtest I did at the time of buffering was on WiFi on my phone.

Things today have been reasonably good. Low level packet loss but only briefly every 3/4 hours that hasn’t caused disruption.

A colleague is going to lend me a Zyxel 5601 fiber router to test. If there is no packet loss on that too, it would appear either the Flint is malfunctioning or it’s being routed differently or something on the VM network.

It’s a weird one. I don’t think it’s Flint software related though. It does it on every single software version I have tried. Openwrt or standard GL-iNet.
 
The big yellow bands on my BQM are new and have only appeared since I rebooted the Super Hub 5 when upgrading the Flints firmware. It doesn't matter what device I use, they're present now and have never been previously.

Show Share Monitor Graph

www.thinkbroadband.com www.thinkbroadband.com

Is it jitter? Is it the router de-prioritising the ICMP request?

As you can see across the top the low level packet loss is present albeit not disruptive.

I have collectd-mod-ping running on the Flint2 today to bbc.co.uk. It'll be interesting to see what the latency graph is like.
 
Last edited:
You must log in or register to reply here.
Back
Top Bottom