The smallest boat I've crossed the channel in was a 35 ft ketch, with 4 mates, admitedly we all have at least Yachtmaster offshore qualifications with about 100,000 sea miles between us!
If you are suitably skilled & kitted (at least charts, tide tables, liferaft, flares, GMDSS radio, lifejackets, epirb, radar reflector, the right clothing, flask with hot drinks/soup if the boat doesn't have cooking facilities - and know how and when to use it properly) plan your crossing well, taking account of the tides and crew ability there's no reason why you can't make a safe crossing even in a small keel boat (say 24ft), personally I'd not want to sail in anything much smaller than 35 ft.
So long as you maintain a good lookout, listen to Ch 16, follow the collision regs i.e. cross any TSS (shipping lanes) at 90 degrees to the big stuff, choosing carefully where you cross between them all then you should have a safe and pleasant crossing.
If you're lucky you'll cross on a nice night with a fullish moon and a good breeze
So basically if you have the skills and kit then there's no reason not to, if you don't then get a ferry instead and leave the coastgard and lifeboat crews safe and warm at home.