I think a more pertinent issue than absolute numbers of people in the country is infrastructure.
If you have ever lived in the south east you'll be aware that the infrastructure is creaking under the strain. Rail and road services in particular cannot cope with an increase in population yet, these are amongst the most desirable places for inward immigration. Water and sewerage are becoming meaningful concerns too. I'm sure the UK is creaking elsewhere but we still continue with our near 20 year policy of what feels like unfettered migration.
There must be billions of people in the World worse off than the majority (or even totality) of UK citizens but we can't heal their woes by inviting them all here. Self interest suggests we should limit migration.
That said, I do not think and have never thought Brexit is the answer to this alone. I would vote to leave on soveriegnty alone, CAP, CFP, application of common market rules to non-common market business and ever increasing euro federalism just reinforce the argument for me.