There are magnets in the currency cassettes, usually 4 magnets.The denomination of the cassette is determined by the arrangement of the magnets polarity and the note configuration in the software. For instance on HBOS atm's a £20 cassette is magnet 1 up and magnet 3 down.
When the wrong cash is given, it's generally either one of a few things. The magnets are set incorrectly for the cassette, the note configuration is incorrect or the staff have put the wrong cash in the wrong cassette. The later happens quite often.
Oh and if cash isn't taken, the dispenser retracts it back in and dumps it in the purge bin.