Of course. Even though I'm a big guy and could probably easily overpower him, and I stay relatively cool under pressure, I still couldn't conceive the idea of being held at knife point by a someone. Whether or not he's a small man is irrelevant - it doesn't take much to inflict a lethal wound with a knife.
He could also run around a bit get you tired out, and bam your dead. Big = slow.