Be prepared for some aggro if you tell him to **** off, if he does just ring the police and say he is aggresive and you feel threatened.
Its upto you OP, how you are going to proceed
Either tell him you want 100mm of concrete (not including hardcore) as agreed and you will be checking the excavated depth, you will be present when they are pouring the concrete, and you want to see a receipt for the concrete to make sure its suitable for a driveway (and not the cheap stuff for backing up kerbs)
if he agrees to this i would let him finish it imo.
but be wary in case half way through pouring the concrete he doesnt come and ask you for extra money to complete the job, excuse like i underestimated the cost and need to add more money to finish the job (travellers call this bouncing a job)
if he disagrees then tell him you are not happy and you dont want him to finish the job, offer him £100 to cover his expenses and be firm but polite.
Threaten him with trading standards and/or police if he refuses to leave.
he wont want them involved as they will ask him for tax numbers/ vat numbers etc....
you are breaking no laws afaik as you are willing to pay for reaonable expenses while he is doing shoddy work.