I have never suffered from this, but my mate did horribly, we would go for a winters morning ride at 8am as the sun would be coming up in February and he would b**** and moan about his bloody neck and fingers...
Anyway, he used all sorts of neck warmers that cost him a truck load (could have braided and brembo'd his crappy GSXR brakes with the amount he spent on neck warmers!).
In the end, as a joke, I bought him a classic knitted scarf from Oxfam for crimbo, bog standard length, pretty chunky, very nice and purpler muhahahaha.
Anyway, he never took the bloody thing off on rides! Said after wrapping it round 3 times, and tucking the scarf down the front, he didn't have any issues at all!!!
So maybe give this a shot?
ags