HOlding down the power button empties any stored power in the PSU - if I kill the mains, and hit the power button, my fans spin up, then die.
Removing the battery means that when you set the CMOS CLEAR jumper to CLEAR or press the CMOS CLEAR button, etc, as I understand it there is a wee NVRAM chip that stores your setup as long as power is applied to it. Remove the battery, it has no power. BIOS clears back to default.