i find it has more to do with familiarity of voices, i used to watch a few anime shows which were subbed and watching the dubbed versions after having listened to them as subbed for so long felt awful as the dubs sounded nothing like the voices i'd become accustomed to.
I think that is a part of it, also it's rather harder to spot a bad performance in a language that the viewer doesn't speak
Oddly enough with anime there have been cases where the Japanese director has looked at the English dub for long running series and said he prefers that to the original (IIRC You're Under Arrest's director is reputed to have told the Japanese VA's that the English ones were closer to what he wanted).
On another odd note, some of the worst "recent" dub decisions for anime have been due to the Japanese companies trying to take an active role in the English dubbing, assuming that what works in Japanese works well in English.
However the biggest problem I think a lot of anime/game Dubs at least for animated materials (be it games, cartoons, films etc) these days have, is that a lot of people remember dubbing as it was done in old days when it often wasn't done with a great deal of care/time spent on it, or much attention to the original script (things like "wordfit"* and spicing up the script, where as these days they tend to work as close to the foreign script as a natural sounding english script will allow).
I think these days it's also not uncommon for the original director/their rep to have to approve dub scripts
Personally I watch a fair bit of dubbed animated stuff, but also tend to watch the original, sometimes I find I like the dub version better than the sub, sometimes it's the other way round (for example I prefer the English version of Hellsing to the Japanese version.
However I must admit I tend to prefer the version I've heard/seen first, as you say you get used to the characters sounding a certain way.
But having said all that, for live action I always watch the original version
