To me, it's really easy to define what scares me in a game as I think about this quite often. It's 90% down to having a fear of not being able to survive.
For instance, early resident evil games scared me and had me on edge as I had limited ammunition and you never knew what was coming around the corner, which had me on the edge of my seat constantly. The same can be said for Doom 3 when it first came out, Dino Crisis and slightly different in Alien isolation, because regular weapons did naff all to the enemy.
Horror experiences are very simple and most game designers over complicate this. It needs to be hard to survive, I don't want to be spoon fed my experience and have the game handed to me. There needs to be planning involved and surprise.
Atmosphere also has a direct impact on that, so I'd say simplicity in keeping things in a survival atmosphere, lack of resources and a decent enough atmosphere to keep the tension up.
Resident evil, Doom 3, Alien Isolation being the top games that kept me on edge. (Dead Space deserves an honorable mention as I loved that game, but it wasn't as scary as it could've been).
edit- Also, I think it has to be said, most horror game fans will say RE1, where the 2 hounds jump through the window when your rushing towards an objective gave everyone the brown trouser treatment. Pretty defining moment in my gaming history.