I was having this same problem, and finally got it resolved. It's not perfect, but I followed the post below, essentially using the various applications (XBMC in my case) to force digital passthrough, while letting the OS go with stereo. Works pretty well.

http://ubuntuforums.org/showpost.php...1&postcount=18

I managed to get it working without removing Pulse. Here is what I did:
Install the Alsa libraries
Open the Pulse default Sound Preferences
Click the Hardware tab
Choose Analog Stereo Duplex profile
In every application that supports hardware configuration, set sound output to IEC985 or similar

Of course your hardware may differ. The surprising thing for me is that if I select any other non-digital profile other than Analog Stereo Duplex it doesn't work. But I now have automatic 5.1 and stereo out working in VLC and XBMC. I haven't spent enough time with MythTV to get that working yet, and I only get stereo from the other media players, but VLC, XBMC and MythTV are the only ones I care about.

My system volume seems to be stable now and not muting on restart, however I also recently switched to Linux Mint (built on Karmic) so I don't know whether that made a difference in that case. I also fixed my VNC problem, though that's OT.