If you don't then some of that ram will go to waste, not that its really needed in linux anyway, but still...
There may be some more teething troubles than with x86 though, can't really speak from experience as i've got 32bit on my laptop for compatibility really