Wiring two amps to one pair of speakers is potential suicide for the amps !, your probably only relying on built in protection circuits if you get away with it, when both get switched on by mistake....
Switch box is one solution. But I use to use the method describe by Mr_S.
Connect a stereo amp to the speakers, connect the L&R pre outs of the AV amp to the AUX input on the stereo amp.
Connect CD player by analogue connection to Stereo amp.
Connect DVD player via SPDIF to AV amp.
To listen to CD just use CDP and stereo amp.
To play DVD switch to AUX input on Stereo amp, and set volume control to about half to 3/4 volume. now use volume on AV amp to control, volume level in the room.
You need to calibrate your AV amp levels with this preset volume position on the stereo amp.
RE AV amps for music. It's more a cost issue then a technical one. It can be done, but of coarse you are paying for all the extra functions and channels, ok if you plan to use them. You are still better to connect a CD player via analogue, and just use the internal DAC's in the amp for DVD's/PC etc.
When you use the analogue inputs, the digital circuits should automatically be shut down or at least out of the signal path, by selecting an analogue input on the controls.
So can AV amp play music... all one boxes, not really, exception that come to mind is the Linn Movie Classic or the NAIM and Arcam equivalent. The combo units from Denon etc struggle if you use the internal power amps. Hook them up to separate power amps, and it gets a lot better.
Processor Pre-amps is where you need to be, like the Tag example. That said I was always told it made a better job of films than music, and still was lacking.
Ok into fanboy mode

the best AV Pre amps are from Linn, these are designed as music playing devises, and when they got that bit how they want, they add more channels of the same quality.... then work out the cost !!
The Linn AV5103, now a bit dated on the AV side, is superb sounding amp in 2 channel stereo, better then the Tag. It's final retail was about £3250, and can be picked up for around 400-700 on auction depending on age and spec.
If you don't need video switching and are happy with DD and DTS decoding only it will spank most stuff. The AV SQ is also better then the new higher tech stuff, which normal sounds harsh and brash at higher volumes.
I sold mine, and bought it's replacement, the Linn KINOS, which again is even better sounding on Stereo and AV, though the AV side is easy not to notice.
It's retail is 4.8K..... there is version without DAC's and video switch etc, and it drops to 3.2K. So as others have stated you pay for the "extra".
For me mine acts as 2 channel pre, a 5.1 channel pre for SACD, DVD decoder, PC decoder for rips, and a multi room driver to feed sound to other rooms, and can be control by my PDA over wireless and LAN