The bit we seem to be missing is why he is saying this. Care to share why he might have this impression?
Stop trying to bait him, It does not matter what the issue is his employer needs to learn some respect. People think because they are manager it means slave driver, And free to humiliate people.
Think again managers because at 5pm you are just normal people easily chinable with a swift hook off the premises.