But by having the windows and doors open all day, you let in 35º+ air... no wonder it can't cool the house.
It's just one room. I've tried multiple methods over the years, had the AC unit a few years now. It's actually far more pleasant now that I've managed to get the old windows to open to 90 degrees and let proper air flow through during the day and only closing up and using AC in the evening.
Previously I'd have had to use AC for most of the day as it would be unbearable. The shell of the house I'm in will heat up regardless. It doesn't take long to get the hot air out of the room, it's that ability to keep it cool because the walls are emitting more heat than usual and the fact that it's still 30degrees outside that's different I believe, as it's not a tight seal to the outdoors due to the nature of it being a portable unit.