It seems clear now that the installation of Font-Manager is what caused this problem. I have not removed Font-Manager since I like it for previewing and hope to be able to keep using it.
So, I am not 100% sure I did this all correctly, but I renamed ~/.fonts.conf to ~/.fonts.conf_bk
~/.fonts.conf looked like this:
It included listings of directories for fonts, but did not include anything in /usr/share so I think that is where the standard fonts are and when the system could not find them, it just grabbed some of the decorative fonts that I had installed into ~/fonts
This file is maintained by Font Manager.
If you wish to make any changes it is suggested you do so using
Any changes made to this file will be automatically relocated there
at startup and any settings already in that file will be overwritten.