I am having a similar problem, but it does start using my swap eventually after a few days of uptime. Gradually day by day it increases until the system swaps itself into an unresponsive state, then I have to reboot. This has happened to me on two separate Ubuntu server virtual machines. One running internally on an ESX 3.5 host, the other running in Amazon EC2.

cat /proc/version
Linux version 3.2.0-25-generic (buildd@crested) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012

cat /etc/debian_version

13:49:30 up 19 days, 20:14, 1 user, load average: 0.33, 0.44, 0.40

free -m
total used free shared buffers cached
Mem: 491 465 26 0 93 202
-/+ buffers/cache: 169 322
Swap: 1715 14 1701

This "inconvenience" has caused me to increase the Ubuntu Server EC2 instance to a m1.large instance size just so it does not use swap. From my experience with Ubuntu Server, while virtualized it is a horrible operating system to rely on. If anyone can point me in the right direction or provide some assistance, I would greatly appreciate it.
Thank you,
Travis Runyard