They can knock on your door, you can tell them to do one, they have no legal authority and if you ask them to leave and they refuse they are then trespassing. If they continue to send you letters or knock on your door you can remove their 'implied rights of access', you can print it off and stick it on your door or just email TV Licensing and request that they acknowledge it, which they always do and then they will no longer contact you and if they do you can then take them to court for harassment.
Worst case scenario, they go to court and get a warrant to carry out a search on your property, what this entails is that they walk in make sure you don't have the TV hooked up.
Of course, you could let them in and they will **** off, however it's the principle.
@the guy above - Nope you don't need it, its only for Television Programs live. And don't forget the 12p
I did the figures a few weeks back, something around £150 pcy.
Edit:
@brucey034 - They used to be able to do it with the old analog signals pretty effectively, I'm really not sure how they do it with the new digital however.