If bufferbloat is all ok, I would then look to see if the network port is flapping.
I assume you have run these:
Is bufferbloat causing issues with your internet connection? Run this test to find out.
www.waveform.com
Use our NEW speed test tool to test how fast your broadband or mobile internet connection really is. Read broadband news, information and join our community
www.dslreports.com
Outside of that, do anything which downloads things like SABnzb etc.. or one else on the network who streams etc...