The problem is whether the packages installed as dependencies of xubuntu-core and xubuntu-desktop are marked as automatically or manually installed. If they are marked as automatically installed the next 'apt autoremove' would get rid of them.You can use 'apt-mark showauto' and 'apt-mark showmanual' with a list of packages to get that status, but typing in that long list of packages is boring and the probability of errors is to high for my taste. I came up with
Code:
apt-mark showauto $(dpkg-query --show -f '${depends} ' xubuntu-desktop xubuntu-core|tr --delete ',')
to automate the process (replace 'showauto' with 'showmanual' to see which packages are not a problem ...). It showed two packages (libu2f-udev and spice-vdagent) which where marked as automatically installed. Running 'apt-mark manual libu2f-udev spice-vdagent' would mark them as manually installed and stop them from getting automatically removed. There might be other packages marked as automatic on your system.
Just out of curiosity: why do you want to remove that font ?
Holger