I have never refunded even part of a deposit, i am totally with Fox and Lucero etc. There is no reason to even consider this, nor consider that this should create any issue at all. A deposit is a deposit, there are no morals to consider. The reason for leaving a deposit and the meaning of the term is very clear.
And on those occasions where the deal did not go through, how did the buyer take it when you told him you wouldn't refund the money?
Assuming this scenario is the same as the OPs - i.e. not at a dealer and the grey area of deposits in which this and many other private sells occur.
We are not all Terry Tibbs.