I'm going to assume you've done all this, but just in case:
In VirtualBox Setting for the VM you have to manually add the USB adapter and also each USB device, it should all show up in the add device. Tehn while running the guest you have to select the USB device from the Machine menu (it's a checkbox). I'm typing this from memeory as I've not used VirtualBox for a while (or even Ubuntu actually ), so it may not exactly match the menus etc.
Almost forgot, and you must have Virtualbox Additions installed on the XP guest.