It really is that simple. As I touched on, as you get bigger your body's hormonal profile changes, and in men certainly testosterone and the ability to store and use muscle decreases (unless you have got a decent amount of muscle mass), and as such it makes it increasingly difficulty to lose weight.
This is why it isn't actually that simple. Hormones such as Ghrelin will inhibit your metabolism making it increasingly difficult to burn the calories the larger you get...it can in severely obese people mean that they are eating very small relative calorie content yet are not losing the weight they should.
I had an interesting conversation about this with a Bariatric Surgeon not long ago and he explained that once the body is past a certain capacity then it becomes virtually impossible to lose weight without intervention of some kind, be that surgery or lipid binding medication.
For the average person who is a few stone overweight then the Calories in...calories out mantra is fine, but in severely obese people the issue is far more complex than simply restricting diets orally.