They shouldn't have charged you it in the first place.
Charges don't magically happen because the bank is in a bad mood, they are an automated result of an account doing something that violates the T&C's that the owner agreed to and constantly agrees to by using the account. Occasionally there are bank errors, however these are such a small minority that they don't really matter.
A bank is within it's right to charge you a £20 overdraft fee for going 10p overdrawn for an hour if that is in the T&C's of the account type, if they say it's going to stand it's going to stand, short of going to the financial ombudsman there is nothing you can do about it. Oddly enough threatening to do that can be a good idea in extreme circumstances as it costs the bank more than you are likely to be getting charged, so it's in their interest to refund them, doesn't always work as it's nearly always a matter of principle over bending to a threat, depends on the manager.
Now that is all simply the way it is, however because generally speaking people who work for banks are human and can empathise, charges that are applied that seem unfair can usually be dealt with.
The problem is this, if a bank says "go £1 overdrawn and we don't charge you", what's to stop someone calling up and saying "I went £1.01 overdrawn and you charge me £20", nothing changes, the only fair way is a blanket "overdrawn = charge" but allow people to challenge a charge they consider unfair.
Every bank charges for violation of T&C's, if a bank allows you to challenge those charges then count yourself lucky.