I usually would have said absolutely. I have a 530d just now, which replaced a fantastic 335d. I loved the 335d, well the engine at least. Mine was "mildly" mapped to 340 bhp, and returned 40 mpg (average, 45+ on a run). It really was so good I didn't need to keep my S4.
However, I am now looking to move on from my 530d as I don't need another big diesel estate, in my current situation. And I am looking for a more "fun" petrol. So I have considered the 340i (the "new" 335i basically).
The thing holding me back, is as Fox has said, the interior. The 5 series has now spoiled me, and sitting in a 3 series just doesn't make sense to me anymore. Swapping to a 535i isn't going to be enough of a change, and being the bigger barge, it simply wont be as "fun" either as its smaller brethren. So I have had to strike off the BMW's from my search (I am currently considering a swap to an older S4 just now, until the new RS4 is released, but I may as well just keep the 530d until then really).