These dont have any baring on the refresh abilities of the panel itself
This is actually wrong, the recent LG 55B6 as tested by AVF recently is 38ms refresh rate , I dont believe a tv has EVER come out as fast as you are stating - most OLED's are in the same 30-40ms range if not more (HDR settings appear to have a adverse affect currently on refresh times)
Most LCD and other tvs have a high input lag... simply because most companies see zero need to put any effort into making them low input lag. It's got nothing to do with the refresh rate.
You can get lets say 1ms response monitor, which has a 10ms input lag, and the same screen used with another controller has 1ms response and 40ms input lag.
TVs are just not great for input lag, or more specifically, their setups and settings are usually a maze of stupidity. Some tvs will have a gaming mode with better settings for input lag, why other modes have higher input lag... who knows. TVs are mostly used for watching, you know, tv, and with a predictable source of information and no need for input it's just not a priority.
Using OLED
TV input lag to compare against monitors and claim the technology is bad is simply flawed as you can see the same increase in input lag between other types of tv panel.
OLED is monumentally ahead of LCD in terms of refresh rate, brightness and many other things. Input lag is almost entirely up to how quickly the controller processes incoming signals, it's controller/electronics/circuitry dependent, not particularly panel dependent. There is literally no reason to ever set up a tv to have higher input lag, there isn't a huge cost or anything else involved, it's just the usual lazy company/designer issue where even though it's stupid and does effect gamers.... they just don't care about optimising it on TVs.