I can sort of see your point - but a lot of the reason I don't have an M5 is because I already have a 530i Sport. It looks the same as an M5, is the same inside as an M5 and I've had it for years and know every inch of it - and I've also kept it in the same condition it was in when I bought it at 4 years old. For 90% of the driving I do (sub 5 mile trips around town in 30 and 40 limits), it is indistinguishable from the M5.
This adds an extra layer of complication - not only does an M5 need to be good enough to justify its costs, but it also needs to be in sufficient condition and deliver sufficient additional satisfaction to justify getting rid of my 530i, which I feel it doesn't.
If somebody ran into the back of my 530i tommorrow and wrote it off, I'd replace it with either an F10 530 or an E39 M5. If I was buying a sub £15k car now and I didn't already have the 530i, I'd be far more tempted by the M3 or M5 given my ultra-low mileage.