It shouldn't kill things if you're only using 20% or so of your upload.
The reason that having a very busy upload will kill your downstream is down to the way TCP, Transmission Control Protocol (of TCP/IP) works. When you get sent a packet of data your PC will confirm to the sender that it was received. If the confirmation is not received by the sender than that packet gets sent again.
If your upload is very busy then the acknowledgement packets don't get through which can result in a huge amount of re-transmission.
Generally speaking, TCP/IP is used when it's important that ALL the data gets through. An email for example, if you're missing some of the data then the email isn't readable. There's also UDP where there is no acknowledgement. UDP is used for VoIP as if a packet of VoIP traffic arrives 1/2 second late then it may as well not bother as the conversation has moved on so it's better to have a very brief dip in the conversation than have part of a word arrive out of sequence.