Well, stop talking smack then
, you know full well that refugees who claim asylum have their movement restricted, here and in the EU. It has always been so. You'd be talking of family reunion in their specific case:
https://www.gov.uk/settlement-refugee-or-humanitarian-protection/family-reunion.
Other countries have similar humanitarian clauses. But even that's not unconditional.
And ordinary non-EU spousal rules have been tightened up via income requirements, etc. This is a non-issue, especially after the crack down on sham marriages in the new immigration bill, which I'm sure you can peruse at your leisure.