Depends on the operating system of the server... there must be a metric somewhere for dropped packets, retransmits, or similar.
Check for duplex mismatches as well - make sure the autonegotiation options on your managed switch's ports match the options/settings on your devices. I've seen some devices switch to really weird defaults when the other side doesn't autoneg.
It's linux so I looked at ifconfig and no errors. Will check the switch too. Looking at it earlier and nothing seems too odd. I can see some CRC errors on one port, so will check that out. It passed the in-built cable test on the switch.
EDIT: So I solved the AirPlay issue by setting the Apple TV (3rd Gen) to stream from the local device rather than from iCloud in the Apple TV settings. Looks like for whatever reason, the Apple TV cannot stream my videos fast enough. Could be my internet connection which isn't that great (20-30 mbps) but I know the videos aren't on my phone either, which is using the same connection to stream them to the Apple TV. Maybe the 3rd Gen Apple TV is reaching its limits - some of these are 4k 60 fps videos, although it can only display up to 1080p.
Also, I tried Plex again on my smart TV, works fine! Maybe what I was seeing before was HDD spin-up on the server causing some kind of streaming lag.
So I've learnt nothing about IGMP but perhaps some of my issues are solved.
The next thing to work out is why is Hive and HomeKit so slow to respond. Maybe I should move all IoT onto the Legacy 2.4GHz SSID? I see some people have an IoT VLAN, I'm not sure if I need to go that far yet. Hive Hub is wired in and almost all else is using it's own protocol rather than WiFi, so not sure what can be done there. I don't have a device to use as a HomeKit hub, not sure if that helps performance there.