Some things that can limit download speed...
The speed at which your line is physically capable of pulling down data.
The speed at which the server is capable/willing to upload data.
Line speed depends on:
the network infrastructure outside of your control (from your house to the rest of the world)
the network infrastructure/ equipment inside your house/ on your premises.
The amount your pay to your ISP.
Server upload speed depends on:
The infrastructure in the place the server is based + the amount they pay to their ISP for bandwidth provision
The amount you pay to the company which owns the server to use their services.
With P2P, like torrents, there are no servers. You are affected by the infrastructure limitations of all the clients, plus any additional limitations such as upload/download ratios.
The question you have asked is a bit general, tbh. Need to know what you're trying to download, and why you think it should be faster...