FTTP VS VIRGIN MEDIA

[Turn]

At least that's powerful, reliable and - most importantly - wired directly into the core switch with no virtualisation nonsense.

Yeah I've got it running on a VM w/ Docker, but ideally I'd do the same. But luckily a beefy server, so the paravirtualization (virtio) doesn't affect things too much afaik.
The setup for SmokePing is way more simple w/ Docker that's for sure, it was a headache setting it up w/o Docker on CentOS awhile back as iirc the documentation wasn't the best.

Thanks again for your settings, I'll match them and compare back with you in a few days/week.

Sounds good, would love to see them stats, let me know if you have any issues w/ changing those settings or setting up the Targets (if you want a cut down version of mine I can provide), & be sure to add fastly.com so it's a fair test
& Just as a FYI if you increase the RRD history to the same size as mine (the bottom half of the Database file). It'll be quite big, as in 140MB for each target w/ my settings as it reserves the space ahead of time.

 

[Rainmaker]

Yeah I've got it running on a VM w/ Docker, but ideally I'd do the same. But luckily a beefy server, so the paravirtualization (virtio) doesn't affect things too much afaik.
The setup for SmokePing is way more simple w/ Docker that's for sure, it was a headache setting it up w/o Docker on CentOS awhile back as iirc the documentation wasn't the best.


Sounds good, would love to see them stats, let me know if you have any issues w/ changing those settings or setting up the Targets (if you want a cut down version of mine I can provide), & be sure to add fastly.com so it's a fair test
& Just as a FYI if you increase the RRD history to the same size as mine (the bottom half of the Database file). It'll be quite big, as in 140MB for each target w/ my settings as it reserves the space ahead of time.

Yeah, I gave up on doing it manually. As you said the project's documentation is massively convoluted, and the freshest guide I could find was 2 major versions out of date and not logically fixable. I've been using BSD and Linux for 20 years but so much stuff has changed in the last 2 versions it's impossible to work out what's what based on either the outdated guide or upstream's documentation. It's a shame because FreeBSD is the nicest server OS I've used to date, for most things, and I run all kinds on it (Signal TLS proxies for people in Ukraine/Iran/Cuba/China, Tor relays and obfs4 bridges, DNS servers, etc). Rather in this case the FreeBSD package is fine, but the paths and configs are unintelligible based on the outdated man from the smokeping project itself.

I'm running linuxserver/smokeping in Docker on my Synology NAS now, and that's very nice and easy to use! I set Targets to include Fastly (as well as my own server). I don't know if you're using the same Docker image, but if not check it out. I like linuxserver's stuff anyway (unless hotio has it), but this is particularly well laid out. They included a DNS menu on the smokeping web page too, which both pings major DNS providers and also DNS queries them all per your settings. I set the queries to match the Database settings for the web hosts (per your config) and it's already looking very interesting!

My own DNS server is as fast as anyone's, which is nice (because it also does ad/tracker blocking and parental controls for the kids). So far, speaking for VM it's what we expected. Your low single digit pings to Fastly's CDN is more like 20+ ms for me, and there's also much jitter; and that's off-peak and with cake enabled! That's on VM's Gig1 with a SH4 in modem mode attached to an x86 router running IPFire. I'll send you a fuller graph in a day/week/whatever you fancy.

 

[Turn]

Yeah, I gave up on doing it manually.

Yeah literally, I gave up on SmokePing the first time cause of the complications with installing it w/o Docker.

I'm running linuxserver/smokeping in Docker on my Synology NAS now

Yeah also using linuxserver/smokeping, and yeah they have a huge collection of useful stuff that's dockerized for you.
I throw all these things hosted things behind a Nginx reverse proxy & use CloudFlare Access (Free) so I can easily access it from anywhere securely.

My own DNS server is as fast as anyone's, which is nice

To be honest my latency is so low to CloudFlare 2-3ms, that I don't even bother with my own DNS that caches requests etc.
I must admit I did try out AdGuard awhile back and it was quite good.

Your low single digit pings to Fastly's CDN is more like 20+ ms for me, and there's also much jitter; and that's off-peak and with cake enabled!

I don't have any queueing rules my MikroTik router to be honest, infact I have fasttrack/fastpath enabled so it just spews out the packets as my pipe is pretty big (940Mbps) and most clients are on the WiFi so it isn't easily saturated anyway.
My location is also pretty good, I'm probably 20-30miles from the PoP in London. Do you get similar latency to CloudFlare & what sort of latency do you get to your first hop when doing a traceroute.

 

[Rainmaker]

Yeah literally, I gave up on SmokePing the first time cause of the complications with installing it w/o Docker.
Yeah also using linuxserver/smokeping, and yeah they have a huge collection of useful stuff that's dockerized for you.
I throw all these things hosted things behind a Nginx reverse proxy & use CloudFlare Access (Free) so I can easily access it from anywhere securely.

Same, I self host everything including Vaulwarden, AdGuard Home DNS, Usenet and torrent box with the *arrs behind Mullvad, Plex, the usual. It's all behind nginx and WireGuard (depending on needs - family can hit the reverse proxy and watch movies, I can VPN into the actual network and admin it remotely). I run a dozen or so servers - Docker, Jails, Linux/FreeBSD/OpenBSD bare metal and more. My traffic is modest, but high for a home user - about 12,000 concurrent connections 24/7, 5TB up and 5TB down each month on average (though that's growing). Hence wanting FTTP so badly.

To be honest my latency is so low to CloudFlare 2-3ms, that I don't even bother with my own DNS that caches requests etc.
I must admit I did try out AdGuard awhile back and it was quite good.

Yeah as above I'm running AGH on Oracle Cloud free tier, so that our whole network is ad/tracker free and it handles parental controls for the kids, neatly wrapped up in DoH and DoT.

I don't have any queueing rules my MikroTik router to be honest, infact I have fasttrack/fastpath enabled so it just spews out the packets as my pipe is pretty big (940Mbps) and most clients are on the WiFi so it isn't easily saturated anyway.
My location is also pretty good, I'm probably 20-30miles from the PoP in London. Do you get similar latency to CloudFlare & what sort of latency do you get to your first hop when doing a traceroute.

Since last night:







OurSecure (oursecure.network) is mine. We've not even gone through a peak time yet, obviously, as the graph only started last night. It'll be good to see a few days' overview to see how things look on a weekly graph. Remember though, icmp/ping sent mostly when your line is idle doesn't tell you that much. It's far more informative to run Flent's rrul test, just in case you didn't know and haven't already. BTW before I go, first hop latency (which is always terrible on VM):

root@freebsd # traceroute 9.9.9.9

traceroute to 9.9.9.9 (9.9.9.9), 64 hops max, 40 byte packets
 1  router (10.100.0.1)  0.237 ms  0.160 ms  0.175 ms
 2  10.53.35.5 (10.53.35.5)  12.441 ms  18.509 ms  17.432 ms
 3  pres-core-2a-xe-116-0.network.virginmedia.net (213.104.74.57)  20.684 ms  12.071 ms  27.584 ms
 4  * * *
 5  * * *
 6  86.85-254-62.static.virginmediabusiness.co.uk (62.254.85.86)  31.502 ms  26.229 ms  29.880 ms
 7  * * *
 8  uk-lon01b-ri1-ae-25-0.aorta.net (84.116.136.102)  30.194 ms
    uk-lon01b-ri1-ae-23-0.aorta.net (84.116.135.30)  26.656 ms
    uk-lon01b-ri1-ae-25-0.aorta.net (84.116.136.102)  23.333 ms
 9  195.66.225.238 (195.66.225.238)  39.628 ms  26.287 ms  33.312 ms
10  dns9.quad9.net (9.9.9.9)  30.065 ms !Z  27.769 ms !Z  26.100 ms !Z

Edit: Notice how my first hop onto the 'public' Internet has an rfc1918 private IP address? Yay Virgin. With BOGONs (private IP addresses) blocked on my firewall from WAN > LAN, I get thousands of hits from that first 'public' VM hop trying to scan my network and access devices. Yeah... no. I posted about it on the VM forums and they 'escalated' it but I never heard back.

 

[Rainmaker]

As an aside, I just took delivery of 2x 16TB WD Red Pro hard drives to upgrade our NAS. The old drives will be sold or repurposed, but for now the easiest way around the upgrade (i.e. clean install on the nas) is to copy their contents to my offsite encrypted pCloud backup. Almost 2TB of files, at 50Mbps. Today I really, really wish I had FTTP.

Edit: I just did the maths to depress myself. On VM at around 51Mbps upload it's 3 days, 23 hours, 51 mins 48 seconds. On gigabit FTTP that drops to 4 hours 53 mins 20 seconds. FML.

@0007 did you see the stats and first hop pings you requested, above?

 

[Turn]

Same, I self host everything.
About 12,000 concurrent connections 24/7, 5TB up and 5TB down each month on average (though that's growing).

Yeah same here, I generally DL 10TB, and UL 1TB / mo.

Yeah as above I'm running AGH on Oracle Cloud free tier, so that our whole network is ad/tracker free and it handles parental controls for the kids, neatly wrapped up in DoH and DoT.

Ah right, I generally just hosted AGH locally on a docker container. But if I were to host it elsewhere I'd probably go with OVH personally as their VPS offering is very cheap.

BTW before I go, first hop latency (which is always terrible on VM):

Here's mine:

As an aside, I just took delivery of 2x 16TB WD Red Pro hard drives to upgrade our NAS.

Oh nice I recently bought a 4TB Seagate Ironwolf drive and it's working nicely. I don't use too much space but yeah that much storage should last awhile depending on how much media etc you hold

Edit: I just did the maths to depress myself. On VM at around 51Mbps upload it's 3 days, 23 hours, 51 mins 48 seconds. On gigabit FTTP that drops to 4 hours 53 mins 20 seconds. FML.

Yeah the upload is really impressive, I rarely use it to be honest, so you'd probably make better use of it than me.

@0007 did you see the stats and first hop pings you requested, above?

Yeah, but it's not the same format or detail as the ones I sent, if you click on each one individually then it'll give you the green graph where the smoke is more clear.
Plus your connection to fastly.com seems a bit flaky, I have 1.1.1.1 setup aswell, so it'd probably be better to compare against that one.

I really need to add some more hosts to be honest, I just set it all up quickly a month or so ago.

@Rainmaker Sorry for the last response.

 

[Rainmaker]

Ah right, I generally just hosted AGH locally on a docker container. But if I were to host it elsewhere I'd probably go with OVH personally as their VPS offering is very cheap.

Did you miss the 'free' in 'free tier'? Can't get cheaper than that. They have nice specs, 2x free x86 VPS and 2x Ampere arm64 VPS (mega fast) with multi-gig WAN. My paid hosted stuff is on Netcup (netcup.eu), they're cheaper than about anyone (£2.50 a month for 2c, 2GB or £5 a month for 4c 4GB, multi-GB WAN and 80TB bandwidth) and they have FreeBSD as well as Linux and Windows as OS options, plus you get FTP and full control of the DVD drive (to upload your own ISOs). I've been really impressed by them, I have a Tor exit node running on one of the 4c/4GB VPS atm under FreeBSD 13 and its been flawless.

Yeah, but it's not the same format or detail as the ones I sent, if you click on each one individually then it'll give you the green graph where the smoke is more clear.
Plus your connection to fastly.com seems a bit flaky, I have 1.1.1.1 setup aswell, so it'd probably be better to compare against that one.

Yeah I didn't even have a full day of data so only sent you the overview. Here's the 3h/30h detail shots:

 

[Turn]

Did you miss the 'free' in 'free tier'? Can't get cheaper than that. They have nice specs, 2x free x86 VPS and 2x Ampere arm64 VPS (mega fast) with multi-gig WAN.

Skim reading never does me any favours
Yeah I've never used them before tbh, but for something super lightweight like AGH it's probably more than sufficient.

My paid hosted stuff is on Netcup (netcup.eu), they're cheaper than about anyone (£2.50 a month for 2c, 2GB or £5 a month for 4c 4GB, multi-GB WAN and 80TB bandwidth) and they have FreeBSD as well as Linux and Windows as OS options, plus you get FTP and full control of the DVD drive (to upload your own ISOs).

That's pretty good, OVH's DDoS protection is very very good, which is why I have always used them.
& Yeah 80TB/mo sounds more than sufficient lol. With OVH, the bandwidth is completely unlimited, but they've made it so the speed of the VPS is based on which plan you get - 250Mbps/500Mbps/1Gbps/2Gbps plans available for the VPS's.

running on one of the 4c/4GB VPS atm under FreeBSD 13 and its been flawless.

I've been using CentOS for awhile now, starting with RHEL6, then RHEL7, and now RHEL8.
But moving towards AlmaLinux because of the mess with CentOS.

Yeah I didn't even have a full day of data so only sent you the overview. Here's the 3h/30h detail shots:

Your SmokePing graph doesn't look great to fastly.com, maybe it's your peering, as to others it looks quite a lot better.
I've just noticed it's a bit difficult to exactly compare as the axis isn't the same, it's probably most appropriate to compare to CloudFlare anyway.
Generally Fastly.com is lower ping & more stable than CF which is why I always use that one to test against.

Cloudflare SmokePing graphs:


You might also notice my graphs have different height/width, that's because you can change it in the smokeping config, in the Presentation file.

 

[Rainmaker]

Skim reading never does me any favours
Yeah I've never used them before tbh, but for something super lightweight like AGH it's probably more than sufficient.

You get 8 fast vcore and 24GB RAM with 200GB SSD storage to divvy up as you like (1x mega VPS, 4x smaller etc), with 5Gbps WAN, free for life. Plus two regular x86 VPS. It's 'sufficient' for AGH in the same way Rolls Royce engines are 'sufficient' to storm down to Lake Garda haha! They're very fast instances, you should sign up (free is free!). The only limitation really is OS, you only get to choose between Ubuntu server or Oracle Linux (RHEL), but that's never bothered me.

I've been using CentOS for awhile now, starting with RHEL6, then RHEL7, and now RHEL8.
But moving towards AlmaLinux because of the mess with CentOS.

When I need Linux on a server, I reach for Alma first. We're in agreement there! I've been using CentOS (and practically every other distro tbf) for the last 20 years. Before CentOS was even a thing, and before Fedora Core (as it was then) was even a twinkle in the eye, I was running Red Hat 2. I did like the early Fedora Core releases, especially 2 and 7. Nowadays I find too many crashes for my liking on a daily driver.

You might also notice my graphs have different height/width, that's because you can change it in the smokeping config, in the Presentation file.

Ah, thanks for the heads up. I pulled 1.25TB down yesterday alone, so my pings are a bit wonky looking today. Nothing terrible mind you, cake is still doing its job on the router. I'm literally willing Openreach to hurry uppppp and fibre my street. Literally two streets away in every direction from my house is now covered. Ironically, our last house got fibre last year, and I'm still kicking myself.

 

[Turn]

You get 8 fast vcore and 24GB RAM with 200GB SSD storage to divvy up as you like (1x mega VPS, 4x smaller etc), with 5Gbps WAN, free for life.

Generally I'm hosting everything locally now, I have a 12600K, 64GB DDR4 server at my disposal, running proxmox.
CPU's generally idle's around 1-2%, so I clearly need to give it something to do

I have a friend who has VM (West Yorkshire), and I ping his IP for him.
Here's the graph of that, and you can kind of compare it to what I've sent above, to see the sort of difference, it's pretty obvious:

 

[Rainmaker]

Generally I'm hosting everything locally now, I have a 12600K, 64GB DDR4 server at my disposal, running proxmox.
CPU's generally idle's around 1-2%, so I clearly need to give it something to do

I have a friend who has VM (West Yorkshire), and I ping his IP for him.
Here's the graph of that, and you can kind of compare it to what I've sent above, to see the sort of difference, it's pretty obvious:

Yeah that's pretty woeful. VM seem to have a default latency around 20ms and it spikes up from there with any activity. On one hand it's amazing how 'blind' to it they seem, it's just brushed away and never really mentioned, except in hushed tones in dark corners. At first glance you'd think latency and bufferbloat weren't even considerations (even some end users say that, as above!). Then you realise DOCSIS by its nature is latent and full of jitter, and that it's literally in VM's interest to play magician and divert the eyes to the headline speeds rather than the murky truth lurking beneath the tablecloth. They can't fix it, it's inherent, so they do everything possible to dance around it.

As for your server, put it to work lad! My house is like a datacentre these days (lol), it's good for the soul. If you're at a loss (and assuming you're already running the *arrs and sab + qbit or whatever), use up some spare capacity and bandwidth running a Tor middle relay. That'll pump your stats haha! If you have a public IP spare run a Signal TLS proxy to help people in (eg) Iran, China, Cuba, Ukraine, Russia etc to communicate freely, too. I run a handful with a successful dedicated Twitter account (successful meaning I have about 100 users in dozens of countries). Here's my *BSD repo, or else Signal have a blog listing how to run it in Docker.

How do you find Proxmox vs (say) Unraid or Dockerising everything? I've never dabbled, but I have the ISO in my NAS. I almost deployed a test instance and reverted to FreeBSD at the last moment (I only really needed a seedbox spinning up, and wireguard-kmod + qbit on a rock solid base was all that I needed really).

 

[ChrisD.]

2x free x86 VPS

Have they discontinued these, or just not available in UK South?

 

[Rainmaker]

Have they discontinued these, or just not available in UK South?

Still available so far as I'm aware, Chris. They do run out of capacity occasionally, but they just add more when that happens. Try a different availability domain - so still in London, but change from AD1 to AD2 or AD3 and you should be good to go. Once they're up you're usually fine.

Edit: Like this @ChrisD. When you hit create instance, expand the 'Placement' section and try one of the other ADs. When one is full usually the others have space. No difference to you, it's basically just saying 'stick it on another rack'.

 

[ChrisD.]

Cheers, going to move my blog over from Lightsail I think.

 

[Rainmaker]

Cheers, going to move my blog over from Lightsail I think.

You can't beat free. They're solid in my experience, albeit with Oracle's clunky way of doing things on the management side (subnets, firewall etc). The Ampere instances are phenomenal, very fast. You can only have four boot volumes across all your free instances, so bear that in mind. What I mean is, you get 2x x64 instances (if you want them, but they're slower) and 'up to' 4 instances of Ampere, but only 4 boot volumes.

So you could do 2x x86 and 2x Ampere, or 4x Ampere, or whatever mix. What you can't do is have 2x x86 plus 4x Ampere - I mean you could, but you'd have no boot volumes/disks for the last 2 as your cap is 4. I hope that makes sense. I ended up with 2x x86 for stuff that has no arm64 software alternative, and 2x Ampere ARM instances for stuff that does (Docker on one, AdGuardHome DNS running on the other).

Once you know what instances you need, you can divvy up the available resources accordingly. On the Ampere side you get 4 cores and 24GB RAM, so for example 2x VPS with 2 cores and 12GB RAM each, or 4x single core VPS with 6GB RAM each etc.

 

[ChrisD.]

I think perhaps new accounts can only have the Ampere instances, which I think will be perfectly fine for my requirements.

I don't get an option for any of the ADs.

 

[Rainmaker]

I think perhaps new accounts can only have the Ampere instances, which I think will be perfectly fine for my requirements.

That's a shame, but as I said the Ampere instances are far superior anyway, and most stuff runs on arm64 these days. I'm sure you're aware ARM is the next 'big thing'TM in server space, so most stuff is ported now.

I don't get an option for any of the ADs.

Genuinely not sure who has misunderstood whom there, but in your screenshot you can choose AD 1, AD 2 or AD 3. When you haven't any availability in one of them (you have AD 2 selected there), one of the others should still allow you to create the new instance. You may have to also edit the shape (eg to select Ampere A1, 2 cores, Ubuntu Server or whatever) to have availability pop up.

 

[ChrisD.]

Genuinely not sure who has misunderstood whom there, but in your screenshot you can choose AD 1, AD 2 or AD 3. When you haven't any availability in one of them (you have AD 2 selected there), one of the others should still allow you to create the new instance. You may have to also edit the shape (eg to select Ampere A1, 2 cores, Ubuntu Server or whatever) to have availability pop up.

The server spec you have in your screenshot, only shows up under legacy and I'm unable to select it. Not too bothered though, I can get what I want out of Ampere, plus, if I need x86 I get $50 a month (I think) Azure credit.

 

[Turn]

You still running it? @Rainmaker

 

[Rainmaker]

You still running it?

Yeah, still plodding along. I only have data from April onwards (when I started), so it might be interesting to see your graph generated from the same time parameters to make it apples to apples. Here's what I have so far: