Hi,
I think you may have missed some basics out.
Can you go into the BIOS and set the following if they are not already set like that;
Advanced BIOS Features -> Full Screen LOGO Show -> [Disabled] This will let you see how fast the RAM and CPU are running
Advanced Chipset Features -> CPU Configuration -> Limit CPUID Maxval -> [Disabled]
Advanced Chipset Features -> CPU Configuration -> CPU Thermal Control -> [Disabled]
Advanced Chipset Features -> CPU Configuration -> C1E Enhanced Halt State -> [Disabled]
Advanced Chipset Features -> CPU Configuration -> Execute Disable Bit ->[Disabled]
Advanced Chipset Features -> CPU Configuration -> Virtualization Technology -> [Disabled]
That lot basically stops any fancy processor energy saving or other features interfere with the overclock. The really important one is the C1E setting as that switches the multiplier down to 6 and drops the voltages to save power when you're not using the machine flat out.
Next,
Advanced Chipset Features -> System Voltages -> CPU Core -> About 1.4V is as much as I would to use initially
Advanced Chipset Features -> System Voltages -> Memory -> 1.9V should be plenty as you only have PC5300 RAM
Advanced Chipset Features -> System Voltages -> NForce SPP -> 1.4V, but be prepared to use 1.5V if you need it later
Advanced Chipset Features -> System Voltages -> CPU FSB -> 1.4V
Advanced Chipset Features -> System Clocks -> PCIe x16_1 -> [100]
Advanced Chipset Features -> System Clocks -> PCIe x16_2 -> [100]
Advanced Chipset Features -> System Clocks -> PCIe x16_3 -> [100]
Advanced Chipset Features -> System Clocks -> MCP<->SPP Ref Clock -> [100]
Advanced Chipset Features -> System Clocks -> nForce SPP-->nForce MCP -> [4x]
Advanced Chipset Features -> System Clocks -> nForce SPP<--nForce MCP -> [4x]
Here we are locking all the system bus speeds so they don't increase when we turn up the FSB on the CPU, which is what happens automatically. We have also locked the Northbridge<>Southbridge communication speed at 4x the speed of the FSB which seems to be important on this board over 1333MHz.
Under Spread Spectrum - Disable EVERYTHING. This is very important.
Spread spectrum is a technology that slightly varies the bus frequency to avoid electronic interference with other devices eg. radios and mobile phones. It also royally messes up overclocking, so it all has to go.
Advanced Chipset Features -> FSB & Memory Config -> SLI-Ready Memory -> [Disabled]
Advanced Chipset Features -> FSB & Memory Config -> FSB & Memory Clock Mode -> [Unlinked]
Advanced Chipset Features -> FSB & Memory Config -> FSB (QDR) MHz -> [1508]
Advanced Chipset Features -> FSB & Memory Config -> MEM (DDR) MHz -> [667]
The SLI-ready memory isn't applicable to you as you don't have SLI RAM, as you only have PC5300 RAM, you need to run unlinked ie. the CPU and RAM run at different speeds. Otherwise, when you increase the CPU FSB, you will also increase the RAM speed. Depending on what RAM you have, this may or may not be a problem. You have Heinz 57 varieties of RAM in different shapes and sizes so it's best to lock it to a slow-ish speed for safety. If you run sync mode you will increase the RAM speed with the CPU and the chances are you'll crash as a result of RAM instability.
You've already found the FSB speed switch
There are a couple of gotchas about the NForce 650 chipset;
1. There are huge FSB holes in the range. It doesn't like 1333-1400 for instance, so sometimes you just have to take big jumps - in this case to 1508. Why 1508? Well, there is a bootstrap switch at 1506, so you want to be on the higher bootstrap, so 1508+ it is.
2. The RAM and CPU speeds are not truly asynchronous. They are actually calculated on a divider automatically when you restart the machine, so watch the RAM speed when you change the FSB speed. You want a round number in both if you can help it, so try and increase the QFSB in steps divisible by 4.
OK - try save and reboot and see what happens. If you don't get any joy at 1508, try 1550 or 1575, you might be pleasantly surprised
Let us know how you get on.