Yeah that is a downside, you'll lose the HD audio, however I went from HD audio on my old xbox one x connected to my amp, to the new series x connected directly to my TV because my TV supports VRR and 120Hz but my amp doesn't, so I'm now having to use normal dolby, and frankly I can't tell the difference!
Losing HD audio on anything but a flagship soundbar will be unnoticed I think. Losing atmos would hurt if you have an atmos soundbar but otherwise optical is fine for all soundbars given their tiny drivers and basic surround setup.