Well the Brexit campaign has been focused on sending money to the EU and not getting representation in return, and doesn't like freedom of movement. So voting to leave and having to keep those in place to secure a trade deal would seem like a hollow victory.
Of course there's also the possibility that a free trade agreement is rejected by the post-Brexit government, having been put there on an anti-immigration / "ARE SOVEREIGNTY" platform, at which point we spend years in negotiations and trade is subjected to tariffs.
The idea that we'd want to actively stop free trade with our closest neighbours in favour of trying to open up further ties with countries half way around the world that have other options for trade is a perverse one. Do you have any more information on how we are restricted with trading outside the EU currently?
This seems to suggest that the 'we can do better outside' argument isn't a valid one.