Difficult isn't it. Yes you can get physical, which can easily end in a charge of assault (especially if they are 15-16 years old) or you can get stabbed, and your bike nicked, depending on where you are in the country.
I would have had the same initial reaction as you, telling them to stop, I would only get physical if I really had to.As above though, your own appearance can have a big bearing on the outcome.If you are an 8 stone weakling dressed in lycra then it's going to be different to a HD riding gorilla covered in tats and beard.