Honestly there isn't a thing about the default install that I do like. Ubuntu still has one of the best repositories and the most ppas and 3rd party support, so that's what I use, but I mostly install everything from the minimal CD or use Xubuntu as a starting base anymore.
I've never liked any of the GUI package managers.
I strongly dislike Unity and find that gnome3 doesn't work well as my main setup as I can sometimes need to have several hundred windows open with almost the same name, which sucks in that environment.
I hate all the search functionality and history in the file system. I don't want my recent documents showing up anywhere. I don't need people to see some racy comic I was reading last evening when I go to open a text document in gedit. I don't need to search my own files; that's just silly.
I'm not a big fan of integration on the desktop and prefer more modular setups like gnome2 or xfce.
I've always disliked the slash screen, but that can be removed.
I've been having a problem with versions 11.x to current getting hardware volume controls to control the correct sound output, but I'm sure I'll solve this once I get a little time to look into it.
The wacom driver in the repositories is still broke as of 12.04, it does not support older tablets like the intuos2 correctly and you have to disable the driver and then load it manually after the system starts to get functionality.
I have a bunch of other nit-picks about different packages, but most of them are simple little things I could compile myself if I really cared.
Right now I'm using some of xfce, compiz, one of the nautilus forks, and a bunch of other stuff and I'm pleased as pie. It just takes me longer to setup correctly whenever I upgrade to a new version then it did five years ago.