How much swap use does free show?
Code:
fred@trusy-ar:~$ free
             total       used       free     shared    buffers     cached
Mem:       8118288    1335480    6782808       9136      78784     527308
-/+ buffers/cache:     729388    7388900
Swap:      1952764          0    1952764
But RAM use is normal. Since most users reuse applications regularly, RAM is cached with what you ran. Only when RAM is full and you load a newer app, then some is released for that app.

Linux ate my RAM! - memory use cache
http://www.linuxatemyram.com/