Of course it is, we have to have necessarily harsher and more retrictive visa rules for non-EU migrants to compensate/offset the numbers coming in from the EU which we can't control.
Despite only comprising of 7% of the global population, EU migration makes up nearly half of all immigration here.
If EU citizens had to face the same visa conditions as everyone else, you could loosen the rules a little (thereby making it easier for Bangladeshi curry chefs to come over) AND reduce the number immigration figure at the same time.
The problem is that for the notion of us being forced to limiting non EU migration (due to the EU being a number that we can't control) to be accurate, it would follow that non-EU migrants are all fulfilling vital high-value roles, which just isn't reflected in the tax contribution figures that compare the two groups.
I believe that the government for years hasn't really been too bothered about migration because it made the headline economy figures look good, and when tackled on it, our EU membership was a convenient bogeyman. Unfortunately when that same government campaigns for remain it tends to smell a bit. I also don't believe that they would be purposely reducing migration in the event that we left the EU.