Smaller/shorter/thinner fan blades need to rotate more times to generate air flow. So higher RPM = more noise, you can get quiet 80mm fans but they don't move a lot of air. 120mm fans are cheap and can get quiet ones which will shift just as much air as 80mm. 140mm fans are great but not everyone can fit them in their case.
In a word, yes. If you purchase a good quiet fan such as the Scythe Gentle Typhoon 1150 you will notice that it is a lot quieter. There are cheaper versions with lower rpm that are just as quiet too. A lot of 120mm fans on OCuK for under £6 which should be far from disgustingly loud. You pay a premium for a fan when they have created a fan which can produce a good amount of airflow to how much noise it creates.