How did you end up with 3 different versions of drivers nvidia-310, nvidia-319-updates, and 331.20? One time I messed up a system trying to switch nvidia drivers using apt-get, ended up with multiple versions, could not purge them properly, and ended up reinstalling Ubuntu. What video card/chip do you have that you feel you need a backported (?) kernel and 331 nvidia drivers?
I am using nvidia-expermental-310 which in 12.04 all shows same nvidia version for each mainstream kernel version for: dkms status
I might have had to use nomodeset while installing 12.04, but not since it was installed (not in GRUB_CMDLINE_LINUX_DEFAULT line).
nvidia-319-updates, 319.32, 3.2.0-54-generic, x86_64: installed
nvidia-319-updates, 319.32, 3.2.0-55-generic, x86_64: installed
nvidia-319-updates, 319.32, 3.2.0-56-generic, x86_64: installed
nvidia-319-updates, 319.32, 3.2.0-57-generic, x86_64: installed