The caffeine has a fairly large effect on the body if taken in large amounts. I would think that it is mostly the effects of caffeine people are feeling. Blood glucose / insulin spikes likely have a part to play too for non-diet drinkers. In 'normal' amounts giving it up likely doesn't have too much of an effect, but for people drinking a litre or more I'd say you might feel the difference.
I class myself as fairly caffeine addicted, I just swapped diet coke for coffee to remove the acid. I'm planning on buying decaf and weaning myself off caffeine a bit. Drinking decaf after 4pm say for a while. My blood pressure is up, and will need treatment if it stays up next checkup. Doctor said lowering caffeine intake should help with it.