In theory at least, 64 bit is slower when working with small numbers, as if 64bit instructions are used, to add 1+1 then its quite likely that it works on a 64bit representation of 1. IE 63 zeros + 1. So thats double the memory bandwidth requirements of 32bit.
Of course properly programmed a 64 bit program should be smart enough to pick appropriate sized registers in the CPU, even in 64bit mode registers as small as 8bit are actually available in the X86 cpu's.
When working with numbers larger than 32bit can handle then 64bit is certainly an order of magnitude faster, but at the moment, the number of 64bit programs that actually do 64bit calcuations is quite small.
In real work, using current applications 64bit is marginally slower than 32bit, unless your heavily into multitasking, and can make use of 64bit's other benifit... the ability to handle more than 4gb of linear memory (IE not using a fudge like PAE)
As long as you have the right drivers, and you dont want to run any 16bit applications, then the downsides of 64bit are quite small. I've been running 64bit successfully now for over a year (Windows XP 64 bit edition). Got drivers for my tomtom, mobile phone, printer etc no problem, and all my applications are compatible. However, if you cant get the drivers, or you have incompatible software then it could be a problem.
32bit has better driver support, is still the 'native platform' for the vast majority of PC software, and is slightly faster in most cases. With the exception I already mentioned, such heavy multi tasking that you need 4gb+ of memory or a single application that would benifit from >2gb memory. Windows 32bit can only assign 2gb of userspace memory to a single application unless the application is coded to directly support PAE.. which is basically Microsoft SQL server, and a few other specialised programs.
For the vast majorty of people, there isnt much difference between 32bit, or 64bit OS.
I've also got 64bit Vista on another computer. It's pretty smooth with 4GB memory installed. Certainly not a problem, although most of the time its no better than the 32bit setup's my friends have.
Good read... im guessing in time 64 bit will be fully utilised, and the on par speed with 32bit will be exceeded... well id think so anyway 