Food for thought: As you like xfce4 ... and have a bit of knowledge of ubuntu;
How bout starting with the minimal install and adding exactly what you want ?
I am running a rather outdated high end box ...started with 13.04 minimal, installed xorg, xfce4 and google-chrome as my base system.
All I can say is : "I like it !"
It is fast, lean and mean - no frills - just what is necessary and only what I want on my operating system.
This route does take being totally aware of what you want your operating system to be.
On the other hand // old Sempron AMD system ubuntu // -> My wife had expressed that only death would separate her from her beloved 10.04 ...flash, facebook and youtube -> dualbooted her into 13.04 lubuntu ...installed google-chrome (pepper for flash) ... introduced her to it ..
much faster as lubuntu 13.04 than ubuntu 10.04 ... she is now a happy camper with lubuntu 13.04 !
There are many many options, depends on hardware and what you want