I say that they will have to use a standard visa process, which isn't "very difficult".
Later this year my girlfriend will move from her student visa onto a partner visa for instance, I know about the application processes, and yes they are annoying, but they don't present such a barrier as to significantly stop applications.
If the UK leaves, UK citizens in rEU, and EU citizens in the UK would have to apply for visas. There will be a transition period of a number of years to allow this to happen. I'm not sure why this is such a difficult concept to you?
To answer your question directly, yes it will be more difficult, we are talking about removing the priciple of free movement afterall, but it will just be a minor inconvience for tourists to and from, and would allow actual government control on migrants to take place.