Ok, its working now using x11vnc instead of vncserver (and running it sudo just incase)
Records movies but the same problem is that it is just too slow to be useful. I recorded at 600x400 res, and it does about 1 frame every 3 seconds or worse.
I gess this nis just a generic problem accessing the memory. Looks like I will have to make some code which saves snapshots ever few frames. and post convert.