The cold, is in the food, the air, has relatively little heat in it to speak of and one pass through the cold side of the refridgeration unit and the air is cold again. The second you open the door the movement of air you create means the majority of the air has been mixed or moved with warm air already. If you close the door in 2 seconds, or 30 seconds, one cycle through the cooling unit and the air is cold again and the actual contents would have barely warmed up at all.
If I won't be long, I'll leave it open for a bit, the act of opening and closing the door causes more cold air to get out and hot air in, than just doing it once most likely.