The RAM and motherboard are working correctly.
You need 2.1-2.3V usually to run PC8000 speeds, but motherboards are set to boot at 1.8V, so the memory manufacturers all got together and decided that they would set their RAM to run at a crappy speed and loose timings (often 667MHz, 5-5-5-15) to make sure they would boot up at 1.8V.
Once you start the machine up, then you can adjust the voltage upwards and increase the speed.
If you haven't increased the voltage, that might explain your BSODs.
Depending on what kind of CPU you have you'll want to vary the multiplier somewhere from 2:1 (2.0) to 3:2 (2.5) and then crank up the CPU FSB.
This is pretty basic stuff and is covered in many DS3 overclocking threads, so you really ought to read at least one of those before you get really frustrated, or worse, potentially damage something expensive.