Xeon vs Ryzen - Dual Xeon is much cheaper than a single Ryzen, and far more powerful, I've read it's hit and miss with 1800x on 4k transcoding.
Those xeons are ancient, and cheap for a reason. Even if your not a fan of Ryzen - build a newer single socket intel based server. Power and noise are a concern with any dual Xeon servers.
hardware acceleration via GPU - I believe the quality is not as good with this? I may try with my current set-up as i just bought a RX580 and see if it helps.
The quality is what you set it at, whether gpu or CPU transcoding. Its always a trade off between speed and quality.
Newer intel chips can transcode using the gpu part via quick sync. Nvidia GPUs via nvenc (but limited to 2 concurrent streams with all geforce cards, and entry level quadring cards). Amd cards can transcode and have no artificial limits, but support can vary.
Even if you could only offload a couple of streams onto the AMD card, that reduces the work the CPU needs to do. (Not sure on Plex, but emby had an option to limit number of transcodes on each form of hardware acceleration e.g. so you could do 2 on an AMD card, 2 on a Nvidia card, 2 via intel gpu, and then anything else would fall back to CPU only transcoding.
transcoding on server vs end player - I'd have to get everyone a nvidia shield or an apple TV
...
PS. also need to transcode 4k to lower bitrate as my internet is crap, so 10mbps 1080p is best I can serve...
Do you have *any* 4k clients? If not then don't bother ripping/sourcing 4k content, or as above do a seperate 4k library, and rerip most common content into a format that doesn't need transcoding and can be direct played at 1080p/720p and a codec that works across your devices.
From my brief trial of Plex a few months ago, I'm sure you can even maintain 1 library, but have e.g. the same film as several different resolution/codec files in each folder - the Plex client picks the appropriate one. I'm sure there's even a scheduled transcoding option to re-encode lower quality copies during off-hours.
By storing multiple copies that don't need transcoding as they stream you reduce the CPU/gpu requirements and push it to storage which is arguably easier to solve.