I got L glass with my 350D, it made a huge difference. Then when I moved to a 5D body, it was nice and everything but the IQ didn't really change, I just got more options. I fully recommend better glass over bodies!
I would, Kind of.
Personally I wouln't go for the consumer body, and then spend glass. I would however spend the money on glass if I had a prosumer level camera instead of spending on the next revision to the model.
i.e. I would upgrade to a 40D from a 350D or even 400D, but I'm not sure I would from a 40 to a 5D or 1D.
My reasoning is that the consumer models are too small, and the thing does not balance well with larger galss. The functions I want are not easy to hand (changing focus points on a 40D is simple and done in about half a second with the camera up to my face, the same operation on a 400D i need to look at the back of the camera). Some of the functions I want are not on the lowest models (spot metering), and I'm not convinced of the drurability of the consumer models.
I have no gripes about the image quality, they seem pretty good from both the 40D and 400D we have (though the 40D probably has the edge).
Mind you, if you are happy with the handling of the camera you have, than by all means spend the cash on quality glass.