TechPowerUp have an explanation of what's going on in the 970:
http://www.techpowerup.com/209339/gtx-970-memory-drama-plot-thickens-nvidia-has-to-revise-specs.html
So basically using the last half GB of RAM is slower if the other 3.5GB is being accessed at the same time (and vice-versa) due to their slightly odd decision to disbale part of the L2 cache. Also Nvidia have knowingly or unknowingly published incorrect specs.
this explains why the fps drops loads and games start to stutter and my work around is to lower down the screen ress of my games to stop it using the extra 500mb of vram