Yes, you are. Buy a 4 Series because you want one and prefer it - not because your Dad has the 5 Series. My Dad has had 5 Series the entire time I've had them - I'm on my third, he's had 4 but has since moved onto a 6 Series. I think they are great cars and don't really understand why people would avoid a car simply because someone they know has one. If you like one, get one, if you don't, don't.
The 640d is simply the better car by absolutely miles. If they are in budget they are massively better than the 4 Series. The interior is exceptionally good, its better specified, its just better. And mechanically its the same as the 4 Series so its no less reliable.
The 6 Series is basically a luxury 5 Series coupe. I'd have had one if they were not so much more money than the equivalent 5 when I got mine (I got a year old 530d for the price of a 3 year old 640d).
The only reason not to buy a 6 Series is if your budget isn't enough for a nice one. They will typically be fairly old, high mileage and tatty if you're looking around the 15k mark.