I don't have time to read the whole thread but I'll chip in my experience...
I've been training in martial arts since I was 6 years old. obviously I've changed a lot physically since then, I'm a lot stronger now.
The moves that they show you do work. The key is making it happen without thinking about it. For example If someone throws a punch at you, you don't really have time to *think* that 'this move' is the best thing to do.
You'll find after a lot of hard work, you won't think that anymore, it'll just happen. Around the house my gf used to hide behind things to scare me (mature I know right

), and my first reaction is instantly to raise my guard and sort out my stance. I've asked her to stop jumping out at (as apposed to just shouting boo) me as I know one day I'll hit her lol.
What I'm trying to say is, it does, after a lot of practice, become second nature. It's at this point that the techniques you have learned become useful.
With regard to training with partners and not injuring them etc... if what you're really after is good defense in an everyday situation, more so than general fitness, then you should look for somewhere which will train you as such.
My instructor I have to say is very good at judging confidence and skill, we worked from slow attacks with indication to no indication, to 100% power with no indication. Yes people got hurt, but at least you didn't get jumped on top of after and battered. We also went from rubber knives, to wooden, to sheathed real, and finally live blade for advanced students.
If you stick at one club you'll find you'll make friends with people and then this aspect of 'Oh my god I hit you int he face as hard as I can' will be alleviated... its laughed off between friends.
I hope I answered some of your questions and doubts
