I would like some OCUK BMW opinions. I'm at the point where I thought I knew what I wanted but I'm now at a loss. I have a F10 LCI 530D I've had it for around 6 years.
Hopefully my opinion can be of value to you as I went through exactly this process around 18 months ago now. At the time I also had a 2015 530d which I'd owned since 2016. It was a fantastic car and in the entire time I had it required only routine servicing and, like you, new brakes all round. Nothing else went wrong, it was perfectly reliable and a great car, so replacing it was always going to be difficult. I considered mostly the same options as you did. Rather than quote each part of your post, I'll just go through each car you've mentioned with my thoughts.
8 Series:
Initially I really did want one of these. I think they look great and the interior is fantastic. I still think this and really the only reason I didn't in the end was that I like to run my cars under warranty and the BMW warranty on the 8 Series was so expensive I'd never be able to justify it, so would have had to run the car without one which wasn't really what I was after. They are also huge cars - which is fine, but puts many off. I still think an 840d is probably one of the best all round cars you can buy, the engine is brilliant. I share your view of the 840i - the engine in these is really very good indeed but alongside the diesel in a large, heavy car primarily used for long trips I've found that it's not the experience you'd neccesarily expect. If I'd have bought an 8 Series, I'd have had another diesel.
5 Series:
The obvious choice. I like the 5 Series a lot and the next generation car was a small improvement on my car in every way. But I wanted something with iDrive 7 which meant a facelift car or one of the very rare last 6 months of the original and they discontinued the 540i saloon at the facelift, so it would have meant a 530d again, as this was the most powerful diesel they offered on UK cars. Which, again, was fine, but I felt like it was time for something a bit different, especially as they'd made a few cost saving corner cuts on the newer car which I think as a direct replacement as a newer, better example of the same thing would have annoyed me after my previous 5 Series. At least with something different you can put these downsides down to the fact it's a different car, so wouldn't neccesarily be as nice as my previous 5 Series.
3 Series:
The current 3 Series, until they facelifted it, is one of the best looking saloon cars BMW have made in my view. They look fantastic and as soon as I saw lots of people complaining it was now so big it was the size of an E39 5 Series, I knew this was an obvious contender. I'd driven several and really liked the way they drove - small enough to be a bit more fun than the 5 Series, but large enough to still be the sort of boring comfortable saloon car I like so much. Sensible warranty and servicing costs and a nice range of options meant the M Performance versions stood out as an obvious consideration, especially after reading reviews where people expected it to be a BMW M3 and then complained it wasn't. Excellent - a 'Performance' 3 Series that wasn't an M3 seemed ideal as I didn't and don't want a BMW M3 and everything that goes with owning one of those. The M340i has almost 50bhp more than the same engine in the 840i and the 540i and this extra power solved much of the 'this is a bit like the diesel only it uses more fuel' issue I had with those. It even made a good noise.
So, I bought a 2021 M340i.
It is not as comfortable as a 5 Series. The ride quality is quite firm, in Comfort mode I find it absolutely fine after I got used to it, but it's not a 5 Series. It's also very refined and nice to drive on the Motorway at 70mph, but again, is not a 5 Series. There are definitive compromises in this area but it's still easily good enough to be an excellent long distance car - the biggest surprise in this area is how exceptionally fuel efficient it is. Over the last 18 months I have averaged exactly 1.5 miles per gallon less than I did with my 530d. On a long Motorway trip, it will reliably return in excess of 45. It's incredible, I've no idea how or why its so efficient but it really does feel like a high performance car with the running costs of a low performance one. I doubt it'd cost me any less to run a 320i. I've not driven an M340d and I understand it is even more fuel efficient, but when the petrol one is this good I'm happy to give up that extra economy for the extra fun of having probably one of the nicest 6 cylinder petrol engines BMW have ever made. Whether it works purely for somebody who likes to only drive up and down Motorways in comfort is another question - for me, its my car for all types of driving.
A note on other things you've mentioned, such as the X5.
During 2021 and into 2022 BMW had some quite significant supply disruptions due to COVID. This is why you are finding X5 without the stereo upgrade and not because owners were deliberately selecting that. Certain items of specification were unavailable across most of the range (You'll find M340i with missing wireless chargers, passenger seat width adjustment and a small number with touch screen issues) or were deliberately held back for M Performance cars only (the upgraded stereo). So, most (every? never seen one that didn't, but always check) M340i, X5 M50i or whatever should have the stereo and the head up display if they have the technology pack but the regular cars may not.
I appreciate that I've not really answered your question here - but I don't think anyone can. It's entirely personal choice which of these excellent cars would work best for you.