That's a likely problem right there.. you will be re-encoding the sequence 3 times each with a lossy codec. A photo copy of a photocopy of a photocopy..... It will degrade at each encode.
Creating high quality video to youtube is a challenge to most, your internet upload connection is likely to be the limiting factor.
Even youtube recommend 50Mbps for a high quality 1080p video. Or 8000kbps for 1080p standard quality.
https://support.google.com/youtube/answer/1722171?hl=en-GB
The problem is that youtube re-encode whatever you send them unfortunately. So unless you record then upload a lossless file your videos most likely will look bad. The Problem with doing that however, is that even with some lossless codec's that enable the file sizes to be a lot smaller than raw, you'l likely be constrained to uploading a 100gb file for a max of 20mins of video if your lucky.
If your not prepared to go that far:
Either upload the avi from afterburner(do it overnight, as your avi may take a long time, plus you shouldn't fall foul to isp limits during this period).
Or not use afterburner, try using either nvidia shadowplay or the ati raptr to capture the gameplay. which create a *264 mp4 using the on board encoders which you can upload. - Youtube will again decimate the quality, however you will only be re-encoding 2 times instead of 3, 1 at recording and 1 from youtube. You can choose bitrates etc from the settings However file sizes should be pretty small in comparison to the avi(afterburner) method.
There are lots of additional things you can do to trick youtube into giving you a better image, such as a very mild film grain or even upscalaing your video to 4k. That said I'd worry about those for a different day.
TLDR... Don't use youtube if you want high quality video, host your 1 time encoded file.