I have this board, gave it about a week on onboard sound then went back to my Xonar DX. Much prefer the sound from it.
Leave it on auto and let XMP do the settings.
And with the DRAM voltage set to 1.35v it is stable?
Maybe its the cpu thats unstable somehow.
Maybe the memory controller needs a bump in voltage.
p.S im still learning too and so dont have all the answers.
There might be a setting in the BIOS (guessing here) that will help.
On Z97 there was a thing called Vrin which if set to 2-2.1v helped a lot, I just cant find anything in the Z170 BIOS that matches this setting.
But have asked around.
Or it could be that with Skylake now has this part given up to the motherboard to control, its no longer relevant for overclocking.
vsa up to 1.5 is fine on air but 1.35 should max out most of the cases
vccio at 1.25-1.35 should also be fine for most of the cases