It's more likely to be the video driver rather than Xorg itself. If there was a generic problem with Xorg then there it would be a big deal and there would be a lot more on it everywhere online, (there may be a big thread on this on the Ubuntu forums but theres nothing on the OpenSUSE or RHEL mailing lists for example).
From other threads you appear to be using the proprietary Nvidia driver which traditionally has gone through times where it hasn't been particularly stable and caused issues. An issue between this and the kernel release which Ubuntu is using is much more likely to be the case assuming that the graphics subsystem is the source of the hangs.
If you did find another display backend then the chances are it wouldn't have the accelerated functionality you have come to expect.