As has been mentioned already; HDMI would be better, if it doesn't cause hassles mentioned above.
With optical; as Esat has pointed out; the audio stream needs to be encoded with Dolby Digital or DTS in order for it to support 5.1. Vast majority of PC games don't have Dolby Digital or DTS, unlike console games which do, so Dolby Digital Live (most common) is required to encode the audio stream with 5.1 in real time. Unless you have a premium motherboard that has it, you'd need to buy a sound card that has DDL.
It will work without DDL though, connecting the AVR to the optical on the motherboard; but you'd be sending stereo to the AVR, which will apply Dolby Pro Logic II to upmix the stereo to 5.1.
Works to fill all speakers with sound, but you won't get the accuracy of sound placement because the incoming audio data is just stereo.
Easier to use HDMI if you can.