You're absolutely right, the bottom line is that the fault is with the government for not planning adequately, not with migrants.
As shown above, net migration is the cause for half the population growth and therefore a substantial amount of the new housing need. Whilst I also don't blame migrants personally (who wouldn't move to a substantially richer country with significantly higher living standards if they could), you can't possibly argue that immigration isn't having a negative effect on the problem (of housing).
You can't just shift all of the blame away from population growth (which can be controlled given the will) to a lack of building new houses (which is a finite solution).
Firstly, let's take your criticism of inadequate planning. How can you plan how many houses you need when you have a rule that says anyone elsewhere in the EU can decide to move to Britain on Wednesday morning and then require a house in Britain on Thursday afternoon?
With "Nationals" (your words, not mine), when they have a baby you have at least 16 years to accommodate the fact they'll need a house given there's a 99% chance they'll live with their parents for that time), that's quite a lot of time to plan. You don't have that time lag to plan when you have uncontrolled immigration. Of course you can mitigate this with immigration predictions but
these have often proven to be inaccurate and are highly reactive to Government policy and economic climate which can change more quickly than you can build houses.
Secondly, this idea we can just build the problem away. At what point does this tactic have to be offset against some form of population control? When we get to 80m, 100m, 200m people in the UK? I'm not suggesting we'll get to those figures but if you hold the view that migration should be ignored in favour of building the given there is a finite amount of land/resources at what point does it become a limit to your outlook?