Agreed but as said, banning manually driven cars is a pretty unlikely scenario. Are you going to remove all the pedestrians, motorbikes, cyclists, workmen, emergency services vehicles as well?
If you have looked at the tesla autopilot it already deals pretty electively with these, there's a video of one taking emergency action after a truck tries to take it off the road.
Most of these autopilots happen to contain multiple cameras and also both infrared and normal light spectrum. This when combined with sonar systems means that a computer can be fully aware of its surroundings all the time something a human driver is unable to do.
As for workmen, when they start building work traffic control can be notified, this will allow redirecting vehicles or produce a a slow calm flow of traffic instead of the start stop we have at the moment.
Emergency vehicles will be easy as all cars will be linked, just upload route and current location to automation control and the vehicles will take avoidance action.
End of the day all the problems you have come up with can be programmed out or systems be installed to tackle the issue and mitigate it.

