It's very easy to set up a direct debit online with just the account number and sort code however it would, I'm sure, be a criminal offence. Also the direct debit guarantee will ensure that your money is returned immediately if anything did happen.
You're perfectly safe in my opinion. After all, you wouldn't think anything of sending a cheque which has exactly the same information printed on it.