Is he OBVIOUSLY a "special ability" case? (Read above, ignore

)
Some men are just short and some are short because of a medical reason.
People in the former tend to have, as already mentioned, Small Man Complex or Little Man Syndrome. Never ever mention it to them!
It's like if you see a girl who is particularly tall say 6ft2+ ...don't EVER say anything no matter how tempted or curious or genuinely interested you are, I almost learnt the hard way!
Basically it wouldn't have helped the situation when talking about being small if you are a particularly tall/built chap. This would have angered him more, by making him think that you are superior just because you are bigger.
In conclusion, it's his problem not yours. You did nothing wrong, although if he is that small you could have probably used a bit of common sense and thought about it a little more before saying anything. You tried to apologise like any normal person after a genuine mistake, but it wasn't well received - not your problem.
It's the sort of "I see a woman with a big fat tummy, do I say congratulations or just avoid the scenario?" question that you should really know how to answer/navigate around.