That wouldn't really be a server as much as a regular desktop with the package OpenSSH-server added, plus maybe a package for VNC or RDP. You'll want to tunnel VNC or RDP over SSH so your machine stays safe. Those protocols are not so secure and tunneling is a work-around to address that.
Another option is just to forward X11 from your remote machine to your local one.
That is the simplest to set up but can be annoying if there is a lot of latency in the connection. Whether you tunnel VNC, RDP or just X11 be sure to use keys for authentication.
ssh -X -i /home/mrwestwood/.ssh/some_key_rsa firstname.lastname@example.org firefox