If using a Live CD or Live USB, you should be able to make a ext2 partition on the USB hard drive and label it casper-rw.
If booting from Live CD press F6 and type <space>persistent.
If booting from Live USB, (made using Startup Disk Creator), you can edit the text.cfg, (or txt.cfg depending on Ubuntu version), thus:
If the Live USB was made using UNetbootin, the file to edit is syslinux.cfg.
append noprompt cdrom-detect/try-usb=true persistent file=/cdrom/preseed/ubuntu.seed boot=casper initrd=/casper/initrd.gz quiet splash --
Things change from version to version so please let me know if this does not work for you.