Default or the 'middle' difficulty for starters, and on subsequent playthroughs or even whilst still on the first go, raise it up, keep it the same, or (can't remember ever having to do this! /toughguy) lower it a notch - all depending on how well I've managed things.
One exception would be a sequel to a game previously completed on highest difficulty. For example, I began ME2 on Veteran...and even found that too easy so I now only play on Insanity. If it's done the same way, then I'm hoping it'll be possible to choose Insanity from the very start on ME3, else I may consider modifying the game files to do so.
And yeah, it also depends on the type of game and what changing the difficulty actually does.