The override is now fixed on the Debian archive side and bullseye installations should work again. Please reopen if you still see reimages failing.

In T374351#10130304, @Clement_Goubert wrote:

Correction, it worked for puppetdb, but they got added back to debmonitor. Will investigate further.

mw2379 is also still in puppetboard: https://puppetboard.wikimedia.org/catalog/mw2379.codfw.wmnet

@Dzahn gerrit1004 is still in puppetdb: https://puppetboard.wikimedia.org/catalog/gerrit1004.wikimedia.org

Something went wrong with the 2430 rename, it's still showing up in Puppetboard: https://puppetboard.wikimedia.org/node/mw2430.codfw.wmnet

I think

I've uploaded a fixed bullseye build to apt.wikimedia.org and upgraded build2001 (the rest of Bullseye hosts is WIP), that unbreak the next docker-report run.

In T372472#10122797, @jhathaway wrote:

Just for fun I tried backporting the patch, and it does apply cleaning, so perhaps running a custom glibc is an option? @MoritzMuehlenhoff we would love to hear your opinion as well?

All done!

I've kicked off the RAID rebuild; it should complete in half an hour. I've also re-added puppetmaster1003 back to active duty.

In T373980#10121443, @fgiunchedi wrote:

Thank you to all involved so far with the reboots -- much appreciated!

I can confirm the hosts are now reachable from alert2002, except lists1004 and lists2001. On these hosts, unlike the others, there is both nft and iptables, similarly there's /etc/ferm and /etc/nftables which I'm assuming is the cause of the problem (i.e. iptables and nftables together). Does that ring a bell @eoghan ?

@VRiley-WMF puppetmaster1003 has been taken out of active duty and I've set downtime, you can proceed with the drive swap any time.

Any host which switches from iptables/ferm to nftables strictly needs a reboot after the provider has been changed. Some of the kernel modules used by iptables cannot be unloaded at runtime without a reboot (I tried various -f hacks, but to no avail). If the old iptables kernel modules are still loaded the constants formerly defined by ferm still persist (and this is what we are seeing here: the hosts don't know about alert2002 being in the new global list of monitoring hosts).

Nice! I suppose the disk swap needs downtime? Then I'll take the server out of rotation Thursday morning (I'm off tomorrow)

Tracking bug is https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1080418

In T373819#10112885, @MoritzMuehlenhoff wrote:

I'll reproduce this in an nspawn contained and report upstream.

In T373819#10112874, @elukey wrote:

Sure, but if it is not d-i that installs it, then it is puppet (via systemd::timesyncd and the related profile included in profile::base) that does it right? If there is a race condition we can fix it in puppet, this is my point, but maybe I am not getting what is your idea/plan for the fix.

I think this might be a bug in the latest systemd update for LTS:

deb11u5 is from the point release, deb11u6 is from https://lists.debian.org/debian-lts-announce/2024/09/msg00001.html (released yesterday)

This got superceded by https://phabricator.wikimedia.org/T328331

The tool has been fixed by https://gerrit.wikimedia.org/r/c/operations/puppet/+/761029, the manager information is now correctly displayed.

This was intended as a workaround for VMs running on Ganeti servers with 1G memory and Java-based workloads which have a lot of memory activity. We started to buy 10G NICs for all server refreshes and at this point (and when the next refresh is done), the old systems should be mostly gone. As such, this is no longer needed.

In T373783#10110623, @elukey wrote:

I updated the bullseye and bookworm netist images this morning

I'll look into a fix

That is already tracked as T348876, I'll merge that in.

In T372507#10093823, @Scott_French wrote:

In any case, while this may(?) be something we can simplify with some package surgery, I'm not sure that's warranted here.

This is something we'll need to think about for any use cases that require co-installable php versions. That clearly does not include the production image use case, but may include, e.g., maintenance hosts (if the migration to mw-script on k8s has not yet completed).

Let's directly install this server with Puppet 7, there should be no issues in the deployment-server manifests in terms of Puppet 5/7 compat at this point.

Anf FYI, https://gerrit.wikimedia.org/g/operations/software/bitu-ldap is a wrapper for simplifying LDAP operations within Wikimedia (originally written for Bitu, but other Python also use it). Should be helpful for writing the dump script.

In T354855#9931347, @Clement_Goubert wrote:

This issue is biting us again, the time between a puppet run that fails to start ferm.service and the subsequent run is enough to cause issues with eventgate, leading to flurries of Can't enqueue job errors in mediawiki. It is in particular triggered when we add new kubernetes nodes, because that causes all other kubernetes nodes in the cluster to update their ferm rules.

Would adding Restart=on-failure to ferm.service for kubernetes worker be a possible short-term solution @MoritzMuehlenhoff ?

Using ldap-maint1001 has the benefit that it already does r/w changes to the r/w slapd servers. Currently we don't restrict that, but we've been gradually shifting r/o access only to the replicas and I'd like to come to a state where the only r/w changes to our LDAP are coming from Horizon (for cloud VPS access management), ldap-maint and Bitu and then all other hosts in production get access denied via firewall rules.

One thing that we could do is to

In T352245#9894693, @Scott_French wrote:

Thanks, @MoritzMuehlenhoff - you're absolutely right that we could decouple these.

The TLS proxy will go away with the v3 migration, since its primary use case will be absorbed into etcd itself (role-based access control). Thus, I think the main question is whether the effort is worth it.

I'll take care of this when I'm back from sabbatical

The old nodes have been decommissioned, all done.

Why bullseye, this should be bookworm? docker-registry is packaged in Debian, so we can simply use bookworm and use the package from it. In fact, we are already using the bookworm package on the existing registry hosts (2.8.2+ds1-1)

New release:
https://lists.debian.org/debian-security-announce/2024/msg00131.html

Indeed, CGO_ENABLED=0 rings a bell.

In T365165#9921708, @Jclark-ctr wrote:

@MoritzMuehlenhoff would you be able to update site.pp file for this server?

The dependency is added because some feature in the compiled Go code uses syscalls which were only wired up in 2.34 (maybe openat() at al). We ran into this problem before and there was a Go build flag to force it to use a fallback. I can't find a reference currently, but maybe Filippo remembers when he's back.

The Buster instances have been removed.

In T367554#9899621, @jbond wrote:

hi all i wanted to say that the sso project is used so that users have an SSO testing infrastructure to use in cloud services. Originally this was also used to provide sso to production like services in cloud services, however this later functionality has been moved.

If there is still a desire to keep a development environment then we will still need all theses machines

• puppetprimary.sso.eqiad1.wikimedia.cloud: The project uses its own puppet master as we have secrets
• sso-db.sso.eqiad1.wikimedia.cloud: This is a mysql db used to store e.g. mfa keys
• sso-pdb.sso.eqiad1.wikimedia.cloud: A puppet db instance, not sure why this is used but guessing the idp classes somehow need some puppetdb functionality.

In T367757#9905191, @Dzahn wrote:

Then the next steps will be creating an SSH key and signing the access agreement. Here are the details:

In T367757#9913302, @kamila wrote:

@KFrancis can you please make sure @MunizaA's NDA is signed? Thank you!

Given that this is a Go static ELF we can also simply build on bookworm and copy over the deb to bullseye-wikimedia, we're doing this for other exporters as well. buster might be tricky due to it's old libc6, but we can also ignore it, there's less than 150 hosts left and they can simply live the old IPMI monitoring.

Very nice!

CAS 7.0 (what we are currently migrating to) removed the memcached backend. As such, this change won't be needed anymore for the idp servers, I'll tick them off.

Did one of these changes possinbly break PCC here?
https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler-test/3739/console

And prior to the migration, puppetserver1001 needs to be allowed in profile::tcpircbot

The task can be closed, or is there anything still open?