No, in general bigger drives are faster. Sometimes the super capacity drives 500gig etc are a tiny bit slower, but all in all bigger=faster if everything else is equal.
Main reason for this, if you use 40gig on a 320gig hard drive, most of the data will be on the outside edge of the disk, this is where the read/write performance is best.
Secondly if your only using a small percentage of the disk, if you keep it defragmented, and compacted, then the average seek times will be a lot lower, as the disk will just be twitching between the various outer tracks. On a smaller disk that was nearly full, it may be making frequent full seeks, from the edge, to the middle, and back all over the place.
Even if you dont need the space, a 'mid range' disk is normally much better value, and will give you more performance than a nearly full small drive.