Usually whatever the default is, for the first go. Rarely have to adjust it before finishing the game once. Then of course often up it for subsequent playthroughs. One exception I can think of is with New Vegas I did Hardcore mode and Hard difficulty right off the bat as I felt like a better challenge and remembered FO3 was mostly too easy. Also when I got ME2, after having completed ME1 on Insanity, set it to Hardcore first and now always Insanity again. Don't think I've ever had to go below the middle difficulty, not that I can recall at least (/toughguy).