What are the output settings on the Xbox set to? And also, what is the capture device capable of accepting as an input? The HDTVtest video posted a page or so ago highlights some of the differences with the output from your Xbox that depend on what settings you choose. This could be causing the banding you see.
output is 4k, 1440p crashes capture device (they think only tv modes matter), I did also try 1080p, but the banding stayed at 1080p.
I tried both 8bit and 10bit colour depth in fidelity settings, and it is using PC-RGB.
I have grabbed another hdmi 2.0 cable from my ps4 pro box and plan to connect it to the console directly to see if things change. The recordings done using the built in feature in my opinion are better quality than the ps4 pro, so I dont mind bypassing the capture device on the Series S. Also I believe if you record HDR content, will colour map it correctly back to SDR on the recordings so can play back fine on SDR devices, something that is not easy to do with OBS.
Personally I think HDR even on something like the LG850 is very noticeable, primarily because you get the much wider colours been used, and properly colour mapped. The problem I believe though is if I remove the capture device, I think HDR will no longer work, because according to displayninja, both the LG and dell nano ips panels cant use HDR on the consoles.
The banding I will say though in practice I think isnt a big deal, I havent noticed banding in the game itself. It seems to be just on that screen dimming.