I used to cycle commute and had this fun.
I found a backpack just made me sweaty and even after a shower I could feel my back clammy. My cyclocross bike didn't have pannier options so I went for an underseat clip-on bag, which I found amazing:
The SQR Slim is a wider version of the SQR Tour Bag and without the external pockets. It fits onto the seatpost of the bike with the Seatpost Quick Release (SQR) system which makes taking the bag on and off simple and easy. This bag was specially designed to fit books, files, notebooks and can...
carradice.co.uk
it leaves a black 2-3" long mount on the seatpost (so I guess it could be an issue if it was carbon, or your stem was slammed) - and the bag just clips on/off this. It's big enough to fit trousers, shirt, a change of undies/socks, a lunchbox, pocket cycling jacket, mini-tool and pump, etc. Didn't go back to a backpack once after moving to this bag - but I'd happily look at panniers if my bike allowed it (and if the bag was easily removeable)
Prep-work helps a lot too.
Leave some shoes at work, so they're left dry and clean. I could shower at work, so I left shower-gel there and took a towel in on mon and home on fri, etc.