It's a potential security risk having potentially vulnerable software on your PC even if it's dormant. But then again IE11 could have a hole which IE10 didn't so does it really matter? No.
IE10 is still being supported by Microsoft.
What security flaws does IE11 have that Chrome/Firefox/Safari/Skype/Steam/Origin and whatever other programs you have don't?
Numerous 0days over the years. The other browsers have been way more proactive when it comes to blocking malicious websites and plug-ins.