Most of the direction issues you speak of are in the set up. I had Chris Franklin at Centre Gravity work on my wife's car 2 weeks after delivery. It fixed the tendency to wonder but what it transformed was the steering which was better weighted and better feel. They do roll a bit and they can wobble too but once you are use to that its a really good road setup. Very adjustable and great fun. I think if you are used to really powerful card you miss a lot of that as you focus on the lack of power. However since ditching my quick stuff I've felt the car is much more fun, though the engine gets loser with miles and is sweeter today than it's ever been. 140mph with the roof down too once though I struggle to remember the last time we were in Germany
I had all kinds of issues with setup on mine, it had been adjusted before (previous owner had lowering springs put on by wheels in motion, and I had it done again when I went for coilovers), but I took it to A-line to get it tweaked and it ended up a right mess. They discovered some of the adjusters were seized and I don't think they wanted to spend the time fiddling with them, and they also left it with the nearside camber adjustment bolt loose - it was seized into the crush tube so I think they'd have had a game to adjust it right, but the overall impression was they'd done half a job and still hadn't got it set up. As they get older the mk3/3.5 will be like the Honda S2000, loads of cars out there that feel a bit funny due to seized adjusters and wonky alignment, but where people don't want to spend the money required to sort it properly.
Ended up taking it to Blink Motorsport to get it properly sorted, remove and replace/grease most of the alignment bolts front and rear, about £300 odd in labour and parts , but it was transformed as you say once aligned properly.
I agree with you on some of the weaknesses - slight lack of chassis stiffness, suspension, and lack of power, I have spent a lot of money on these things and its improved it loads (chassis bracing, coilovers/ARBs and geo, and engine mods to ~200bhp inc lightened flywheel + other bits), definitely the most fun car I've had. Can't see me changing it for a while given running costs and reliability vs fun.