Associate
Quite a lot of potential reasons - nVidia drivers do quite a bit of stuff in software which will use more RAM including hooking/intercepting DX API functions and forced threading models, etc. there is also some difference with the way shader caching (and probably other caching works) where stuff is kept resident in memory even if its not being used unless a low memory situation is encountered where it can be freed up to try and alleviate that.
But wouldnt that be in vram not system ram, and although primitive discard isnt enabled yet primitive shaders apparently is. But i saw one video which was running a 7700k and it was 64 vs 1080 in witcher 3, the ram usage difference was around 1k of ram constantly