My problem with immigration is that it lets criminals in as well, and when they're in, even if they offend it's near impossible to get rid of them.
How big a problem is that really? This is a high profile one I agree, barely a week goes past without an outraged Daily Mail article about somebody who we can't deport, but what are the real numbers?
I doubt criminals would have any real issue with simply entering illegally anyway. By their very nature they have demonstrated that the law isn't really something they have much respect for. Plenty of countries have criminals illegally present there. Not sure how leaving the EU would really make much of a difference in this regard.