I have found the M4 to be weak after sticking by it for ages, I've now moved over to the scar with a red-dot and noticed people just go down easier. I was even running FMJ on the m4.
It's the 20 damage at range, you really need to use it with stopping power just like the ACR (which I'll be switching to
once I have wrapped up 250 headshots on the M4A1).