Comes down to how well the plastic is finished/designed. If not done to a good standard then plastic isn't much, if any more durable than anodized aluminium i.e. just look at mrk's note II where he dropped it a few times, has a dent/crack look (similar to aluminium), then the amount of people who have complained about scratches on their GS 3 and note II back plastic covers etc.
If aluminium isn't finished properly then it will suck but if done very well then it is pretty damn durable against scratches etc. Just watch videos on youtube where the htc one gets scratched by keys etc. you can rub the marks of and it looks brand new again.
For example my one S has a few nicks from a waist height drop onto bumpy concrete, however, the plastic trim around the glass screen looks much worse than the aluminium body (only a bit of the MAO layer came of, which is easily fixed with a sharpie pen

)
I don't see how people who want a premium/more solid feeling device are "muppets"
Plus as mrk said, premium doesn't = heavy i.e. one s is 119g and iphone 5 is 112g iirc but they still feel a lot more premium/solid than most heavy plastic phones.
At the end of the day, the most important thing (screen) will most likely always break/crack regardless of the type of material used for the casing when dropped on concrete. Hopefully the rumours about this unbreakable screen are true.....
They can keep the back plastic for all I care, just make it more premium looking like the Moto X back.
+1
No problem with plastic whatsoever as shown by the likes of the one x, moto x, lumia phones.
The only problem I have with Samsung etc. phones is the finish of the plastic i.e. the pattern look with a glossy smooth finish, just doesn't feel that nice to touch, fingerprints and grease galore + builds up grub more easily than a textured matte back or aluminium casing I find.