benchmarks are utterly useless when it comes to real world usage, just epeen numbersi.e. GS 4 when it first came out was the highest scoring device but in real world usage was super laggy.
Nexus 4 is still one of the fastest and smoothest devices out there, the only device that I would say is ever so slightly quicker etc. is the HTC one, although with 4.3 now it probably just edges the htc one (haven't used a nexus 4 running 4.3 yet)
In all fair honesty mate, a few of my friends have a HTC One and have had a play with my S4 when I had the latest stock running and they said it was just as smooth and slick as their HTC One. I think the original stock firmwares were as what you described, but to be honest it's very good now.