I find this thread very interesting because I have exactly the same problem. I never used to have, it's only now that I am a little older (35) and it's come on in the last five years or so.
As Raikiri said "Nothing to do with confidence, as someone said earlier it's natural to dislike eye contact. Unless you're in competition with the person, which I suppose would be the equivalent to a fight."
I am totally confident (I think) and I don't have a problem per se communicating with people but I *am* aware I find it hard to keep eye contact and I am aware others notice it. We are even talking my mum, my fiancee (wife tomorrow) and my colleagues/boss at work. I consciously try to keep eye contact but because it is a conscious effort it's like keeping a muscle tensed and after a while the effort is too much.
A wierd thing though - I have no problem with people wearing glasses :/
I dislike it if someone artificially keeps eye contact or tries *that* stare - at that point I will stare them out and I don't believe I've ever been stared down.. that isn't a confidence issue.
General, subcinscious social eye contact can be hard and it's really worries me that a lot of people in my life have noticed and seem not to try to keep eye contact with me.. as I said, even my other half which I feel is really important
To those who "mock", that's fair enough but it's a serious issue and not everyone is comfortable with it - your problem if you are insecure enough not to be comfortable communicating with someone who doesn't keep eye contact. There are a lot better/more effective ways of reading communication between people.