NO.
If anything, the men used to wielding weapons and wearing heavy armour since the age of 10 (or thereabouts) would be even bigger.
From digging up the remains of gladiators in Italy, they have discovered that their arm/shoulder bones (size and density) were 4-5 times or more greater than a 'normal' man. With the muscle that comes from building up this density and carrying very heavy weapons you would expect them to be enormous. Probably bigger than most of the people on the GOT tv show.
Not so sure. Sure might have high bone density (rugby players have high bone density from repeat impacts for example - would imagine the impact of practise weapons would be similar), doesn't mean they'll have the massive bulk. You have plenty of folk these days who do manual work day in day out and don't look anything like hulking bodybuilders.
Just looking at suits of armour from a few hundred years ago and the average height was at least a foot lower than now too - mostly I assume to do with diet.