When I think about this I look at it with equivalence. If I was to strike my 4y/o son the equivalent would be some guy around 7ft and built like an Olympic power lifter giving me a half hearted back hander, not good. Now thinking in those terms were a man of that size to lower his voice and instead of using his hands simply tell me in no uncertain terms I was out of order that would do the job.
I only have to threaten sanctions or lower my tone to get my children into line. My oldest son can be a bit of a problem due to his size 6-1"ish but I have that nailed too. He will goad me and just laugh off what I say thinking he can get away with things. I ignore him and simply tell him a sanction is coming, if he taunts me enough it's gloves off time. When he will not back down I pin him to the floor and rub my knuckles in his ribs (not to hurt but to tickle and annoy) he backs down and makes his apologies pretty quickly.
He has actually told me he does it on purpose some times to to get a play fight out of me because he misses being younger and doing that kind of thing more.