We are running into an issue where we have a package being held back from an automated call to `apt-get dist-upgrade` that we do on boot, it breaks upon simply adding the dependency of `chrony`.

  • How can we fix the issue across the fleet with the automated install?


The package we’re installing is a custom meta package which we host on our aptly server.

  • Its sole purpose is to provide a list of other utility and tooling packages to install for our application. Examples include: bolt, can-utils, cpufrequtils, crda, cron, , and several others, etc. This has worked fine for years.
  • We usually never have an issue with installing new utility or any packages by adding them to the Depends portion of the debian/control file, but we are running into this issue when adding `chrony` as a Depends.



This is a similar issue:

  • https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2016141
  • In the linked bug they describe the issue being an upgrade to 245.4-4ubuntu3.21 / 245.4-4ubuntu3.20
  • We believe our problem is slightly different than theirs simply because we use systemd/focal,now 245.4-4ubuntu3.15 amd64 [installed,automatic]
  • We also believe we have a slightly nuanced requirement to ideally avoid updating systemd or any other packages aside from chrony; reason for this is that we have many servers in production which are controlling safety critical robots and would like to minimize unforeseen library interactions if possible.



We think the issue has to do with the fact that chrony replaces/conflicts/provides time-daemon services:


Code:
Conflicts: ntp, time-daemon
Code:
 Replaces: time-daemon

 Provides: time-daemon
And that time-daemon is currently provided by systemd-timesyncd. Which is where our problems start.

These might also provide some relevant information:

  • https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=902026
  • https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=895821


Output of Solutions we've tried :
When we only have the debian/control file set the Priority field set to required and add chrony as a Depends we see :

Code:
server@server:~$ sudo apt-get -o Debug::pkgProblemResolver=yes dist-upgrade --dry-run
Code:
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Starting pkgProblemResolver with broken count: 2
Starting 2 pkgProblemResolver with broken count: 2
Investigating (0) systemd-timesyncd:amd64 < 245.4-4ubuntu3.15 @ii mK Ib >
Broken systemd-timesyncd:amd64 Conflicts on time-daemon:amd64 < none @un H >
  Conflicts//Breaks against version 1:6.2p3-4 for openntpd but that is not InstVer, ignoring
  Conflicts//Breaks against version 1.1.8+dfsg1-4build1 for ntpsec but that is not InstVer, ignoring
  Conflicts//Breaks against version 1:4.2.8p12+dfsg-3ubuntu4.20.04.1 for ntp but that is not InstVer, ignoring
  Considering chrony:amd64 3 as a solution to systemd-timesyncd:amd64 17
  Added chrony:amd64 to the remove list
  Conflicts//Breaks against version 2.1.8-focal.20231103.154153 for CUSTOM-meta but that is not InstVer, ignoring
  Fixing systemd-timesyncd:amd64 via keep of chrony:amd64
Investigating (0) CUSTOM-meta:amd64 < 2.1.7-focal.20231025.184228 -> 2.1.8-focal.20231103.194012 @ii umU Ib >
Broken CUSTOM-meta:amd64 Depends on chrony:amd64 < none | 3.5-6ubuntu6.2 @un uH >
  Considering chrony:amd64 3 as a solution to CUSTOM-meta:amd64 4
  Holding Back CUSTOM-meta:amd64 rather than change chrony:amd64
 Try to Re-Instate (1) CUSTOM-meta:amd64
Done
Calculating upgrade... Done
The following packages have been kept back:
  CUSTOM-meta
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

We tried copying the provides/conflicts statements to our CUSTOM-meta:
And then installing. Also does not work. Here is the relevant debug output:

Code:
server@server:~$ sudo apt-get -o Debug::pkgProblemResolver=yes dist-upgrade --dry-run
Code:
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Starting pkgProblemResolver with broken count: 2
Starting 2 pkgProblemResolver with broken count: 2
Investigating (0) systemd-timesyncd:amd64 < 245.4-4ubuntu3.15 @ii mK Ib >
Broken systemd-timesyncd:amd64 Conflicts on time-daemon:amd64 < none @un H >
  Conflicts//Breaks against version 1:6.2p3-4 for openntpd but that is not InstVer, ignoring
  Conflicts//Breaks against version 1.1.8+dfsg1-4build1 for ntpsec but that is not InstVer, ignoring
  Conflicts//Breaks against version 1:4.2.8p12+dfsg-3ubuntu4.20.04.1 for ntp but that is not InstVer, ignoring
  Conflicts//Breaks against version 3.5-6ubuntu6.2 for chrony but that is not InstVer, ignoring
  Considering CUSTOM-meta:amd64 4 as a solution to systemd-timesyncd:amd64 17
  Added CUSTOM-meta:amd64 to the remove list
  Fixing systemd-timesyncd:amd64 via keep of CUSTOM-meta:amd64
 Try to Re-Instate (0) CUSTOM-meta:amd64
Done
Calculating upgrade... Done
The following packages have been kept back:
  CUSTOM-meta
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.




Background :


We maintain a fleet of Ubuntu 20.04 servers which uses a custom script to update download and then in a separate step install packages.
The command we use to perform installation of the updated is :

Code:
apt-get -y dist-upgrade --no-download


Note: the reason for the `--no-download` is we already download them to speed up the installation process.

From the debian/control file the package in question is currently installed as

Code:
Section: embedded
Code:
Priority: optional


Our hypothesis is that the issue is stemming from the fact that we are adding `chrony` as a new `Depends` and during the install process `chrony` will replace `systemd-timesyncd`


What we’ve tried :



In order to test the above hypothesis we have tried updating the debian/control file for our custom package to look like :

Code:
...
Section: embedded
Priority: required
…
Depends:
…
Chrony,
…
# We tried to match Chrony’s debian control fields for Conflicts, Provides, and Replaces https://salsa.debian.org/debian/chrony/-/blob/debian/latest/debian/control#L37
Conflicts: time-daemon
Provides: time-daemon
Replaces: time-daemon
Regardless of whether we match chrony's `Conflicts, Provides, Replaces` -- If we install the packages manually then we see that chrony and the custom package are installed without issue.

Here we demonstrate it with a dry-run command

Code:
chomper@GZDEV22:~$ sudo apt-get install -oDebug::pkgProblemResolver=1 CUSTOM-meta --dry-run
Code:
Reading package lists... Done
Building dependency tree      
Reading state information... Done
Starting pkgProblemResolver with broken count: 1
Starting 2 pkgProblemResolver with broken count: 1
Investigating (0) CUSTOM-meta:amd64 < 2.1.7-focal.20231030.141854 -> 2.1.8-focal.20231103.154153 @ii pumU Ib >
Broken CUSTOM-meta:amd64 Depends on chrony:amd64 < none | 3.5-6ubuntu6.2 @un uH >
  Considering chrony:amd64 1 as a solution to CUSTOM-meta:amd64 10004
  Re-Instated chrony:amd64
Broken CUSTOM-meta:amd64 Conflicts on time-daemon:amd64 < none @un H >
  Conflicts//Breaks against version 245.4-4ubuntu3.15 for systemd-timesyncd but that is not InstVer, ignoring
  Conflicts//Breaks against version 1:6.2p3-4 for openntpd but that is not InstVer, ignoring
  Conflicts//Breaks against version 1.1.8+dfsg1-4build1 for ntpsec but that is not InstVer, ignoring
  Conflicts//Breaks against version 1:4.2.8p12+dfsg-3ubuntu4.20.04.1 for ntp but that is not InstVer, ignoring
  Considering chrony:amd64 1 as a solution to CUSTOM-meta:amd64 10004
  Added chrony:amd64 to the remove list
  Fixing CUSTOM-meta:amd64 via remove of chrony:amd64
Investigating (1) CUSTOM-meta:amd64 < 2.1.7-focal.20231030.141854 -> 2.1.8-focal.20231103.154153 @ii pumU Ib >
Broken CUSTOM-meta:amd64 Depends on chrony:amd64 < none | 3.5-6ubuntu6.2 @un uH >
  Considering chrony:amd64 1 as a solution to CUSTOM-meta:amd64 10004
  Considering maas:amd64 0 as a solution to CUSTOM-meta:amd64 10004
  Re-Instated squashfs-tools:amd64
  Re-Instated snapd:amd64
  Re-Instated maas:amd64
Done
The following additional packages will be installed:
  maas snapd squashfs-tools
Suggested packages:
  zenity | kdialog
The following packages will be REMOVED:
  systemd-timesyncd
The following NEW packages will be installed:
  maas snapd squashfs-tools
The following packages will be upgraded:
  CUSTOM-meta
1 upgraded, 3 newly installed, 1 to remove and 0 not upgraded.
Inst squashfs-tools (1:4.4-1ubuntu0.3 ubuntu-experimental focal:focal [amd64])
Inst snapd (2.51.1+20.04ubuntu2 ubuntu-experimental focal:focal [amd64])
Conf squashfs-tools (1:4.4-1ubuntu0.3 ubuntu-experimental focal:focal [amd64])
Conf snapd (2.51.1+20.04ubuntu2 ubuntu-experimental focal:focal [amd64])

Inst maas (1:0.7 ubuntu-experimental focal:focal [all])
Remv systemd-timesyncd [245.4-4ubuntu3.15] [systemd:amd64 ]
Inst CUSTOM-meta [2.1.7-focal.20231030.141854] (2.1.8-focal.20231103.154153 CUSTOM-experimental focal:focal [amd64])
Conf maas (1:0.7 ubuntu-experimental focal:focal [all])

Conf CUSTOM-meta (2.1.8-focal.20231103.154153 CUSTOM-experimental focal:focal [amd64])