I use SoundSource and have the same problem. Had an email from the devs saying that version 7 is in beta at the moment which hopefully fixes it.

Several enhancements have been made for Bluetooth devices.
- Multiple changes have been made to reduce glitches and dropouts.
- Device iconography has been updated for several Apple and Beats devices, including AirPods 4 (with ANC), and Beats Pill.
- Unnecessary drift correction is no longer applied to Bluetooth devices, resulting in fewer glitches.
- SoundSource now only shows Bluetooth devices that are either already connected, or available to be connected to, rather than all paired devices.
Enhancements have been made for AirPlay devices:
- AirPlay devices re-connect more reliably after waking the Mac from sleep, as well as on launch of SoundSource.
- SoundSource now automatically reconnects to AirPlay devices when network topology changes, such as those which occur when connecting to a VPN, are detected.
- AirPlay device audio-visual sync has been improved.