Pay the money, don't go through insurance. It will end up costing you in the longer term for what is a paltry sum.
I really am struggling to visualise your description of events. Why was he trying to pass your car in an area well away from other cars, was he trying to park? Was he on the roadway, or in the parking lines? You should have been aware of other vehicles. How did the vehicles hit when you were reversing, you don't say whether you reversed into him, or side impact etc.
To me, it sounds like you assumed nothing was behind you because you went what you thought was an isolated area, and you just reversed without looking.
What is worse, is that you have made an agreement, had time for him to get quotes etc, and it's come back pretty darn cheap and now you are looking to renege on that?