I swear it never used to, but I’ve noticed the same, even when the X isn’t doing anything in the background. Doesn’t make a huge difference having it on power saving mode other than about 5 seconds more startup time and no remote features.
I enabled instant on and made sure nothing was running. Fan is no longer running when off now. In my case I bet it’s Destiny 2 still running.