The first time I had to give up on a game because it was too scary was playing AvP as a marine.
Dead Space is kind of scary but to me it's more about the tension. When I play it I find I'm creeping forward slowly with my gun constantly held out, just waiting for the next attack. It's unnerving, but nothing compared to...
Amnesia: The Dark Descent. Just the scariest game I've ever played, by a country mile. Before I gave up on it, I was literally playing in ten minute chunks. I'd go ages without so much as seeing anything hostile, but it was still unbearable. I had a close call with one of those...things once and it was easily the scariest gaming moment of my life. I wish I could go back and finish it but I just can't.