Some time ago ATI introduced memory error correction routines on their cards, meaning that instead of going all BSOD, producing artifacts or crashing the display driver, the card will quietly repeat failed actions multiple times until it works. What this means in practice is that at some point in your overclocking you will actually start to lose performance even though the card seems stable at the higher clock, this is why running benchmarks is much more important nowadays to see whether the performance gains are legitimate. You need to find the sweetspot, a balance between memory and core overclocks before them routines kick in, start by keeping the core and reducing memory.