I didn't actually think about driver updates since I rarely play games that require them. I'll have to update my AMD drivers and see if anything changes.Weirdly there might be a memory leak at 4K which isn't happening at 1440p for some reason but not sure it is the game itself - I've been playing about a bit and I can play for hours at 1440p without any performance issues but 4K is degrading over time slightly but that is carrying over to desktop even after I quit the game the mouse feels like 30Hz on desktop and you can see it isn't moving smoothly and other games have compromised performance even after a forced GPU driver restart, weird.
I am on 576.02 drivers though:
"The NVIDIA 576.02 driver, while generally stable, has been reported to have a significant memory leak issue, leading to performance issues and crashes, especially during extended gaming sessions."
So I probably should stick the hot fix on there and try again. Fortunately I've not had any of the driver issues with my 4080 Super which is plaguing some 5000 series owners.
The behaviour you have described is exactly what I am seeing as well.

