I've spent two months in Chiang Mai and if I were you I'd definitely try to get it in, even if you're just there for 4-5 days. You don't have to spend a huge amount of money either, you can get a 'decent' room in the centre of Chiang Mai for about ฿100 a night, have an elephant riding+trekking day for ฿600-700 and maybe try a cooking class for a few hundred too, and it's all good fun. Definitely head up to the north to get a different experience to the standard BKK, pattaya, southern islands affair.
Who am I to lecture on such things, but if you are going as a backpacker, don't help to perpetuate the stereotypes of them; walking through the streets swigging 40oz's, shouting like monkeys, going about topless etc. A lot of backpackers seem to think 'being in thailand' is a license to go wild, because the Thai people don't mind it!
In reality they do, whilst they might not say it to your face, they've certainly told me how they feel. Change the paradigm!