The answer to this question depends on what the car is and how much it costs. Generally speaking a well looked after modern car that's done mostly longer journeys will be in as good if not better condition than one with half the mileage which has had a far more average usage pattern. My own car is still absolutely immaculate despite very high mileage. Therefore, logic dictates that high mileage is perfectly fine.
HOWEVER..
This becomes a little more blurred when you are buying a more complicated car. In this circumstance, often the ability to have a warranty on the car will save you more money and give you more peace of mind than buying a higher mileage car which somebody won't sell you a warranty for. It's all about jumping through the hoops of the warranty firms - I doubt the higher mileage car is any more prone to failiure, but they beleive it is so you won't get a warranty on a 100k mile car so might instead be forced to buy a much lower mileage car to ensure you can buy a decent, high quality warranty for it.
But for something like an 04 Mazda6 for which this doesn't apply, buy the best condition one you can find.