[go: up one dir, main page]
Page MenuHomePhabricator
Create Task
Maniphest T373819

Issues reimaging kubernetes workers due to user conflicts in systemd-timesyncd
Closed, ResolvedPublic
Actions

Description

It seems that hosts are having an issue installing systemd-timesyncd=247.3-7+deb11u6, possibly due to a difference in script behaviours between deb11u5 and deb11u6. During a puppet run post-imaging, the user systemd-timesync is being created with uid 999, which based on our adduser configuration is not a system user, and so the installation fails. I'm not certain that the package is at fault here, but the timing of this package release makes me suspicious - servers imaged last week with the old version did not have this issue.

For a host that received deb11u5:

hnowlan@wikikube-worker2050:~$ id systemd-timesync
uid=105(systemd-timesync) gid=111(systemd-timesync) groups=111(systemd-timesync)

For a new host trying to install deb11u6:

hnowlan@wikikube-worker2070:~$ id systemd-timesync
uid=999(systemd-timesync) gid=999(systemd-timesync) groups=999(systemd-timesync)

Sort term, this can be hacked around by running sudo userdel systemd-timesync && sudo run-puppet-agent

Package failing to install via puppet:

Notice: /Stage[main]/Systemd::Timesyncd/File[/etc/systemd/timesyncd.conf]: Dependency Package[systemd-timesyncd] has failures: true                                                        [37/997]
Warning: /Stage[main]/Systemd::Timesyncd/File[/etc/systemd/timesyncd.conf]: Skipping because of failed dependencies
Warning: /Stage[main]/Systemd::Timesyncd/Service[systemd-timesyncd]: Skipping because of failed dependencies
Error: Execution of '/usr/bin/apt-get -y -q remove --purge apt-listchanges' returned 100: Reading package lists...
Building dependency tree...                                                                      
Reading state information...                                                                                                                                                                       
The following package was automatically installed and is no longer required:
  python3-debconf                
Use 'sudo apt autoremove' to remove it.                                                          
The following packages will be REMOVED:                                                          
  apt-listchanges*                                                                               
su: warning: cannot change directory to /nonexistent: No such file or directory
ERROR:debmonitor:Failed to execute DebMonitor CLI: 'NoneType' object has no attribute 'source_name'  
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up systemd-timesyncd (247.3-7+deb11u6) ...                                                                                                                                                 adduser: The user `systemd-timesync' already exists, but is not a system user. Exiting.
dpkg: error processing package systemd-timesyncd (--configure):
 installed systemd-timesyncd package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:                                                        
 systemd-timesyncd
E: Sub-process /usr/bin/dpkg returned an error code (1)

Related Objects
Search...

Event Timeline

hnowlan created this task.Mon, Sep 2, 5:07 PM
Restricted Application removed a project: Patch-For-Review. · View Herald TranscriptMon, Sep 2, 5:07 PM
hnowlan removed projects: netops, SRE.Mon, Sep 2, 5:08 PM
hnowlan updated the task description. (Show Details)
hnowlan removed subscribers: • ayounsi, cmooney.
hnowlan updated the task description. (Show Details)Mon, Sep 2, 5:10 PM
JMeybohm subscribed.Tue, Sep 3, 9:10 AM

I don't see anything obvious in the diff of those two packages.
The systems prior to yesterday seem to have installed systemd-timesyncd during d-i, whereas the new ones did not. There the user creation is the first thing logged in syslog (so right after d-i?).

hnowlan renamed this task from Issues reimaging kubernetes workers due to package user issues in systemd-timesyncd to Issues reimaging kubernetes workers due to user conflicts in systemd-timesyncd .Tue, Sep 3, 9:40 AM
hnowlan added a comment.Tue, Sep 3, 9:54 AM
In T373819#10111943, @JMeybohm wrote:

I don't see anything obvious in the diff of those two packages.
The systems prior to yesterday seem to have installed systemd-timesyncd during d-i, whereas the new ones did not. There the user creation is the first thing logged in syslog (so right after d-i?).

Agreed that there's nothing in the diff so this isn't the package's fault most likely, and the uid range being used is what the package is configured to use. Good point on the user being created first - this happens well before we configure adduser.conf so the uid used is a valid system user up until we configure that file via puppet.

JMeybohm added a comment.Tue, Sep 3, 10:04 AM
In T373819#10112192, @hnowlan wrote:

[...] - this happens well before we configure adduser.conf so the uid used is a valid system user up until we configure that file via puppet.

But that is also true for when the package is installed during d-i (where out adduser.conf is also not in place). But maybe it's a totally different story in case of d-i. I wonder why tasksel does no longer select the package. It seems to be the only one missing...

Clement_Goubert added a subscriber: elukey.Tue, Sep 3, 10:53 AM

There was a point release yesterday which could explain why it changed cc @elukey

elukey added a comment.Tue, Sep 3, 12:41 PM

The new point release for Bullseye is https://www.debian.org/News/2024/2024083102, and indeed I updated the netinst image yesterday.

Could it be a matter of having an explicit ordering dependency between profile::adduser and profile::systemd::timesyncd in profile::base ? IIUC now the user is not installed anymore during d-i, and somehow there is a race between adduser.conf and systemd-timesyncd being created.

MoritzMuehlenhoff subscribed.Tue, Sep 3, 12:48 PM

deb11u5 is from the point release, deb11u6 is from https://lists.debian.org/debian-lts-announce/2024/09/msg00001.html (released yesterday)

JMeybohm added a comment.Tue, Sep 3, 1:03 PM
In T373819#10112747, @elukey wrote:

The new point release for Bullseye is https://www.debian.org/News/2024/2024083102, and indeed I updated the netinst image yesterday.

Could it be a matter of having an explicit ordering dependency between profile::adduser and profile::systemd::timesyncd in profile::base ? IIUC now the user is not installed anymore during d-i, and somehow there is a race between adduser.conf and systemd-timesyncd being created.

Just to be precise: The whole package (systemd-timesync) is not installed anymore during d-i, which is probably because of the new netinst image.

elukey added a comment.Tue, Sep 3, 1:10 PM
In T373819#10112849, @JMeybohm wrote:
In T373819#10112747, @elukey wrote:

The new point release for Bullseye is https://www.debian.org/News/2024/2024083102, and indeed I updated the netinst image yesterday.

Could it be a matter of having an explicit ordering dependency between profile::adduser and profile::systemd::timesyncd in profile::base ? IIUC now the user is not installed anymore during d-i, and somehow there is a race between adduser.conf and systemd-timesyncd being created.

Just to be precise: The whole package (systemd-timesync) is not installed anymore during d-i, which is probably because of the new netinst image.

Sure, but if it is not d-i that installs it, then it is puppet (via systemd::timesyncd and the related profile included in profile::base) that does it right? If there is a race condition we can fix it in puppet, this is my point, but maybe I am not getting what is your idea/plan for the fix.

MoritzMuehlenhoff added a comment.Tue, Sep 3, 1:12 PM

I think this might be a bug in the latest systemd update for LTS:

  • systemd-timesyncd deb11u4 and deb11u5 have Priority: standard, i.e. they get automatically installed by debootstrap
  • deb11u6 however as released by the latest LTS security update has Priority: optional, i.e. it doesn't get installed by debootstrap.

So installations which happened on Monday before https://lists.debian.org/debian-lts-announce/2024/09/msg00001.html went out, should have worked fine, but once deb11u6 was on security.debian.org the new Priority took effect and made it no longer being installed.

I'll reproduce this in an nspawn contained and report upstream.

MoritzMuehlenhoff added a comment.Tue, Sep 3, 1:25 PM
In T373819#10112874, @elukey wrote:

Sure, but if it is not d-i that installs it, then it is puppet (via systemd::timesyncd and the related profile included in profile::base) that does it right? If there is a race condition we can fix it in puppet, this is my point, but maybe I am not getting what is your idea/plan for the fix.

For explicit Puppet conditional we need to consider two cases:

  • On Bullseye the systemd-timesyncd user is created by adduser, so we need to depend on File['/etc/adduser.conf'] (the puppetised adduser.conf)
  • On Bookworm the system user is created by systemd-sysuser, so we need to depend on Systemd::Sysuser['/etc/sysusers.d/sysusers-base-config.conf']
MoritzMuehlenhoff added a comment.Tue, Sep 3, 2:10 PM
In T373819#10112885, @MoritzMuehlenhoff wrote:

I'll reproduce this in an nspawn contained and report upstream.

This has been tracked down following my report on IRC: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1003949 changed the Priority for systemd-timesyncd for bullseye, but that override is missing for bullseye-security. This will need to be fixed by the Debian archive team (FTP masters).

MoritzMuehlenhoff added a comment.Tue, Sep 3, 3:08 PM

Tracking bug is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1080418

akosiaris awarded a token.Tue, Sep 3, 3:15 PM
joanna_borun triaged this task as Medium priority.Mon, Sep 9, 2:34 PM
MoritzMuehlenhoff closed this task as Resolved.Tue, Sep 10, 5:17 AM
MoritzMuehlenhoff claimed this task.

The override is now fixed on the Debian archive side and bullseye installations should work again. Please reopen if you still see reimages failing.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits