Well, I've migrated from Maths (undergrad degree) to mathematical engineering (my masters was in aeronautical engineering and PhD doing numerical analysis in a mechanical engineering dept).
I can tell you without a doubt that the general quality in engineering departments is severely lacking in comparison to math departments. I've seen this trend at three different Universities (Cambridge, Imperial and Nottingham), so I have a fairly wide frame of reference.
I know this won't be a popular point of view, but it's well founded and generally accepted as true (in academia at least).
Anyway, in terms of sheer workload I would have to go for medicine or vetinary medicine as being the 'hardest' degree. Sure, maths is a very hard degree and probably accessible to a smaller number of people than medicine, but if you're smart enough you can get away with doing only a moderate amount of work. Not so with medicine.