Refs do a generally decent job in a high pressure, extremely fast moving environment where they are expected to see every little thing. The situation is further exacerbated by players who, rather than being honest and assist the ref, attempt to con him every moment of the game. Further add to this (understandably) partisan fans who will call for everything, managers who will call for everything and then dogs abuse to top it all off.
Until players play fair, managers call it fair and fans don't call for everything then mistakes are almost inevitable.
If you want nigh on perfection then be ready to accept 3 hour long NFL style matches with constant stoppages, reviews and a generally tedious experience.
In the current setup mistakes are and will remain a permanent part of the game.
As an aside to what someone mentioned, pipeline, we will never have that in the UK. You're expected to start at the bottom as a ref and the grief they get is appaling. They get abused verbally, threatened regularly with physical violence and intimidated most games by "fans" and they have zero protection. Until we (the Brits) impart a bit of decency and treat them even if not with respect bit like human beings you will never get people wanting to come through. They then hit the top and get abuse on a national scale?! It's a mugs game.
The FA should stop paying lip service to respect and make it so, red card instantly if you lip off to the ref. Give it one or two weeks it will stop, over time the respect (that sadly is enforced because most footballers seemingly have no decency) will filter down and you will get more refs coming through, better standard etc etc etc.
TLDR; **** refs? Blame the players and the fans first.
