It could be that the combination of fans I have is causing turbulence, but I'm literally seeing 15C rise ( from 35ish to 50+) when I put the side panel on.The Q300L has a reputation for poor thermals even though it seems like it has a lot of ventilation.
My old 2600x (95W) with stock cooler in a Q300L didn't hit the 40/50C with no load that you're seeing though so I'm not entirely sure it's a case issue. Try 1 intake 1 exhaust for now?