I am having a similar problem, but it does start using my swap eventually after a few days of uptime. Gradually day by day it increases until the system swaps itself into an unresponsive state, then I have to reboot. This has happened to me on two separate Ubuntu server virtual machines. One running internally on an ESX 3.5 host, the other running in Amazon EC2.
Linux version 3.2.0-25-generic (buildd@crested) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012
13:49:30 up 19 days, 20:14, 1 user, load average: 0.33, 0.44, 0.40
total used free shared buffers cached
Mem: 491 465 26 0 93 202
-/+ buffers/cache: 169 322
Swap: 1715 14 1701
This "inconvenience" has caused me to increase the Ubuntu Server EC2 instance to a m1.large instance size just so it does not use swap. From my experience with Ubuntu Server, while virtualized it is a horrible operating system to rely on. If anyone can point me in the right direction or provide some assistance, I would greatly appreciate it.