Whats the ID.
My advice would be to make a small jig.
10mm threaded bar will fit into the drill. If it will also go into the ID of the tube then all you need to do is make a spacer With a 10mm hole and either a lip or taper to the true ID of the tube and use bolts to hold the tapers to the tube.
It's called a spool clamp. And if all you are doing is polishing then you should be fine. You may want to make a support for the free end.
http://www.hereticalbuilders.com/showthread.php?t=52
The cylinder/tube is only 5'' long but 3'' OD and I need to sand it down in dia but evenly.
Ask a local machine shop if they'll do it