We might be experiencing the same issue. I had no problems until I upgraded from 12.04 to 12.10.
It just stalls on the same short fragment for while and than move on.
However, I am using regular Ubuntu with Unity not Xubuntu.
Assuming it's an issue with the alsa driver since, like you, the problem occurs in all media players including flash player, and html5 video on youtube.
➜ dpkg -l | grep alsa
ii alsa-base 1.0.25+dfsg-0ubuntu3 all ALSA driver configuration files
ii alsa-utils 1.0.25-3ubuntu2 amd64 Utilities for configuring and using ALSA
ii bluez-alsa:amd64 4.101-0ubuntu6 amd64 Bluetooth ALSA support
ii bluez-alsa:i386 4.101-0ubuntu6 i386 Bluetooth ALSA support
ii gstreamer0.10-alsa:amd64 0.10.36-1ubuntu1 amd64 GStreamer plugin for ALSA