I'll admit I started off with much the same opinion as the OP, but really that's just casual prejudice because I hadn't really thought much about it.
The fitness side of this I'm fine with, where it's just muscle tone - I'd like to look like that myself.
The bodybuilding side where blokes are just 'bulky' I don't like.
I reckon it's because physically they are taking up more of the room, I sort of find it intrusive in the same way I don't want to sit next to a 20 stone person on a bus. The prejudice that they are also a bit dim makes me think I'm unlikely to find them interesting (some people think fat people are dim too)
It's also a subliminal threat, I don't want to be around someone who could probably toss me out of a window. I feel the same way about people with blackbelts in TinkyTonkydo. I know the wouldn't actually do anything, but the fact that they could, does make me feel differently about them, it's a sense in male inequality that somehow I don't feel if someone if cleverer or better looking than me.
My main issue is that it's all a bit vain, it doesn't really seem to be about fitness but about being bigger than the next guy - as such I associate vanity with unmanliness, I'd feel the same way about a bloke with a ring in his ear, it's just unnecessary.
Because of the work I do I'm putting on muscle, on one hand it's nice to be able to do things without tiring, on the other hand I don't want to look like a bodybuilder.
In other words, you feel less of a man when they are in the room.
Doesn't make them gay, makes you a wimp.