Try apt full-upgrade instead of plain apt upgrade

The problem with both apt upgrade and apt-get --with-new-pkgs upgrade is both of those will install new packages,
but neither can remove packages...