I use Gentoo largely for the same reasons. I still do see a few noticeable performance increases, though, python, boot times, etc. among them.

The benchmarks I found seem to suggest that Gentoo does make a reasonable difference.