i agree, start from the command line and add only what you need.
with debian use the net installer, uncheck everything at package selection.
with ubuntu use the alternate installer, select install a command line system.
once you have the base you can make it anything you want. a custom install is the best way to get the most speed.
Bookmarks