What makes you associate 'plastic' with 'crap'? I suggest you go on a two day plastics course as I did, and learn about the vast range of plastics and their applications. The polymer selection process and engineering that goes into the single back cover of a smartphone is nothing to scoff at.
If anything, you have fallen for the marketing because I'm pretty confident that, overall, the plastic that Samsung use in the S4 will have more favourable properties than the aluminium used in the HTC One or the iPhone 5. For starters, it'll be far more sustainable...
I'd guess at the aluminium used in the construction of these to just as easily be recycled as the plastic could be in the S4. It will also be much more conductive for the heat produced by the phones, that's before how much more premium (IMO) both the iPhone 5 & HTC One look and feel over the S4.
The plastic used may not be simple, but it's nothing like the well finished aluminium of the other two.