Don't bother with mounts. The trade off is nowhere near worth it. It cuts all party banter, as said above it drops out of gallop all the time, and you can't go 5 metres in some maps without an enemy appearing and attacking which means getting off, fighting, the mounts disappears, and you have to call it again.
I only use the mount when I have to return to an area I've completed and I want to travel quickly. Otherwise, as you say, it just has too many negatives.