I've driven an A3 3.2 V6 and an A5 3.0 TDI Quattro. True, the S3 isn't an A3, so you'll need to bear that in mind.
The A5 is a more modern car inside than the A3 - as you'd expect given it came out some 5 years after the A3 first appeared. It therefore has a more modern and better specified infotainment system than the A3. It's also got a more 'contemporary' looking interior.
The A5 is bigger and heavier than the A3 and the 2.0TFSI in the A5 is only available with about 210bhp max - not the 270bhp of the A3. To get a more powerful petrol A5 means finding a V6 - I have never seen one, not sure anyone bought it - or going for the S5. The S5 is a V8 only until very recently and the V6 petrol Audi's are notorious for a thirst that doesn't really match the performance they offer.
So, the 2.0TFSI is probably the petrol engine to go for in both A5 and A3 - and it'll be swifter in the S3 than the A5.
Sadly both cars were pretty underwhelming for me. I have this sort of love/hate thing going on with Audi. I really like the exterior styling - at times. The A5 is a bit superbland but generally speaking Audi get styling right more often than BMW do. They tend to look more 'premium' outside than the equivilent BMW. This is perhaps more of a criticism of how un-premium the previous 3 Series looked I guess, but there we go.
But then you get inside - where everyone bangs on about how amazing Audi quality is - and you find rattly creaky armrests and suspiciously cheap and tatty plastics. Perhaps I set my standards for them too high and am then dissapointed but my standards are set based on the reputation Audi seems to have in this area. The A6 is another story but certainly the cheaper Audi's do not quite feel as 'really nicely made' as perhaps people will have you beleive.
We'll move onto the drive.
A3 - the biggest problem here was that it felt nose heavy but this is a specific problem with the V6. The S3 is mechanically very similar (Quattro aside) from the Golf GTI which didnt have the same problem so I'd not expec it from an S3 either. The steering lacked any real feel, but it was otherwise ok.
A5. Just dull really. It was a diesel - but it was the 3.0 TDI which is supposed to do 0-60 in 5.9 seconds. Perhaps they are 5.9 moon-seconds or something, I dunno. Again, same lack of any real feeling from the steering.
All this leads me to think that really Audi is best suited to people who simply want 'Some nice posh car'. An Audi will look good on your drive, will be reasonably well made and gets you from A to B as quickly as you want given your budget. Thats it. No more garnish, or sauce - thats what it does. If a Ford Mondeo or a Mazda6 is the automotive equivilent of a fridge then an Audi is a really posh stainless steel fridge.
Before I get jumped on for trolling the Audi thread - which genuinelly isnt my intention - there is not neccesarily anything wrong with being an automotive stainless steel fridge. Infact just LOOk at Audi sales. Its what people want. Most people don't care about balance, steering feedback, blah blah blah. They just want a car thats nice to sit in and looks good. This is why you see so many A3 2.0 TDI S-Lines even though the design has been around since the old king died.
If thats you - go for it. And you'll probably love it.
I have never driven any of the really highly rated Audi's such as the RS4 and I very much doubt the above applies to those. Its just my thoughts on the normal Audi's I've driven.
Audi's are nice cars. I think that sums it up really.