I'm exactly the same in the evening, I don't generally bonk as a rule, but it always seems to be very grindy on the way back. At first I thought it was elevation on the route- there's a significant, but not huge, net elevation gain on the way back, and in addition it's mainly sharp uphills and long gentle downhills on the way in in the morning- great for maintaining pace but not so good on the way back. But the main, and surprising thing I found was one day the ride in was hard, and the ride back was great and I did it in record time. Couldn't work out why until I checked the weather- it was an 8mph North-Easterly tailwind on the way back, very rare in Surrey where we generally get 5-10mph SouthWesterlies. Made a huge difference.
Although I still can't work out why it always seems to be a headwind in Richmond Park, no matter which way you go....
Also bear in mind that spending a day at work does take it out of you, even if you're doing a desk job.