I have a USB headset device with a mic and speaker built in. I have my settings configured to properly accept input from the device, and it works fine. However, I would like all audio output from Skype (or any other specific application) to route to the speaker in this headset, while allowing everything else to sound off on my default speakers.

Yes, the audio does work with the mic, but it takes all audio output to it, and not just a specific application.

I expect this to probably be something to do with PulseAudio, because in the skype settings, the only option is "PusleAudio Server (local)", and no real USB Devices.