He moved out of the room on purpose to get someone else (you) to move in because they knew you'd be more likely to move in to the larger room.
There's no way in hell I'd swap now. There's no need to feel bad, they tried to dupe you into moving into a room knowing they'd try and swap you out of it. They didn't tell you at the time you took the room it was adraw every so often to see who gets what room, if anyones been ***** it's them.
They did a dodgy deal to get someone in (presumably to save them having to spilt the rent 3 ways instead of 4). They can argue amongst themselves who takes the small room, it's got stuff all to do with you. They did the deal not you.
Don't take any crap, they tried to be smartarses and knew what they were doing, you did not. Nothing to do with you whatsoever no matter when you said anything about it or not.
If they say anyhting about the draw you can tell them you'd already told them you came into the house on the basis of the room you have, you've already told them you're not swapping around in the past and if push comes to shove they been listening to you so you couldn't be arsed to argue about the draw but you're still not moving.
Sounds grumpy i know but people trying to dupe others like that gives me the right hump, especially if they try to guilt/bully you into it.
Unless they are thick they are perfectly aware what they are doing and that it's not on, especially as the small room bloke got left out of the draw so it wasn't even a fair chance all round.
Stand your ground, remind them it was a deal between the three of them, not you, no hard feelings, but it ain't going to happen.
Unless of course you said, yep, lets do a fair draw in which case Duhhh!