(Go: >> BACK << -|- >> HOME <<)
Page MenuHomePhabricator
Create Task
Maniphest T334517

upgrade contint servers to bullseye
Closed, ResolvedPublic
Actions

Description

The CI servers (Puppet role(ci)) are running Debian Buster which rea and must be upgraded to Bullseye. As of April 2024 the hosts are:

  • contint2002.wikimedia.org | Zuul, Zuul merger, Jenkins, Jenkins agent
  • contint1002.wikimedia.org | Zuul merger, Jenkins agent

There are dependencies before the upgrade can happen, notably Zuul requires python2.7 which is no more officially supported on Bullseye and is only included for the purpose of building Chromium.

Runbook

The reimaging of the two hosts is done in three phases:

  1. Reimage contint1002
  2. Switch over services from contint1002 to contint2002
  3. Reimage contint2002

1) reimage contint1002.wikimedia.org

We need to bring down the two services on the host, reimage it and bring back the two services. The host is:

  • attached as a Jenkins agent to the Jenkins controller which runs on the other host
  • running the secondary zuul-merger daemon (and its companion git-daemon)

Disable services

  • Disable the zuul-merger on contint1002 by setting profile::zuul::merger::enable: false. That should stop and mask the service. There is another Zuul merger system running on contint2002.wikimedia.org.
  • Run the host down cookbook to disable monitoring and alarms

Reimage

  • Reimage contint1002 to Bullseye. Data in /srv can be wiped out, they are merely used for caching (git repos, docker images and build layers)
  • While cookbook is still running but host is already back up and ssh access has been restored.. manually run "sudo a2dismod mpm_event" and run puppet again. Cookbook should now detect a succesful puppet run and finish cleanly.

Enable services

After host is back and provisioned, verify:

  • /srv is a standalone partition!
  • Docker daemon is started.
  • Zuul has been deployed (not by Puppet): /srv/deployment/zuul/venv/bin/zuul-merger. - FAILED
  • git-daemon is up (systemctl status git-daemon).

Enable the services:

  • Enable the Jenkins agent via https://integration.wikimedia.org/ci/computer/contint1002/ the ssh host key would need to be verified again since the reimaging causes the host key to change.
  • Set profile::zuul::merger::enable: true. Running Puppet will unmask it and start the service. It logs in /var/log/zuul/merger.log.

2) Switch over services

Before reimaging contint2002, we need its services to be moved to the reimage contint1002.

Before the maintenance

  • Disable the zuul-merger on contint2002 by setting profile::zuul::merger::enable: false. That should stop and mask the service. There is another Zuul merger system running on contint1002.wikimedia.org.
  • Clean up some of the Jenkins artifacts to reduce the amount of data that will be transfered
Rsync data and states

Synchronize data and states to pre warm the other host:

  • sudo rsync -ap --whole-file --delete-delay --info=progress2 /srv/jenkins/ rsync://contint1002.wikimedia.org/ci--srv-jenkins-
  • sudo rsync -ap --whole-file --delete-delay --info=progress2 /var/lib/jenkins/ rsync://contint1002.wikimedia.org/ci--var-lib-jenkins-
  • sudo rsync -ap --whole-file --delete-delay --info=progress2 /var/lib/zuul/ rsync://contint1002.wikimedia.org/ci--var-lib-zuul-

Switch over

  • Downtime both contint2002 and contint1002
  • Disable Puppet
  • Stop the services sudo systemctl stop jenkins and sudo systemctl stop zuul
Rsync data and states

Now that services are stopped, resynchronize all artifacts and states:

  • sudo rsync -ap --whole-file --delete-delay --info=progress2 /srv/jenkins/ rsync://contint1002.wikimedia.org/ci--srv-jenkins-
  • sudo rsync -ap --whole-file --delete-delay --info=progress2 /var/lib/jenkins/ rsync://contint1002.wikimedia.org/ci--var-lib-jenkins-
  • sudo rsync -ap --whole-file --delete-delay --info=progress2 /var/lib/zuul/ rsync://contint1002.wikimedia.org/ci--var-lib-zuul-
change DNS
  • Change contint.wikimedia.org CNAME from contint2002.wikimedia.org to contint1002.wikimedia.org
change primary host in Puppet/Hiera/CI config
  • profile::ci::manager_host: contint1002.wikimedia.org
  • In profile::zuul::merger::conf change gearman_server to the IP of contint1002.wikimedia.org: 208.80.153.39
  • Run Puppet on contint1002 to point the zuul-merger to the new host
Start services
  • Update Zuul config: from integration/config: ./fab deploy_zuul
  • Enable and run Puppet on contint1002 which should bring up both Jenkins and Zuul

Verify:

3) reimage contint2002.wikimedia.org

TODO copy paste 3) reimage contint1002.wikimedia.org checklist here.

References

Details

SubjectRepoBranchLines +/-
role: delete ci_test role, not used anymoreoperations/puppetproduction+0 -73
contint: enable zuul-merger daemon on contint2002operations/puppetproduction+0 -1
hieradata: delete host contint1003operations/puppetproduction+0 -5
site: remove contint1003 - former testing machineoperations/puppetproduction+0 -4
ci: set puppet7 at role leveloperations/puppetproduction+3 -2
contint: set new default docker version for bullseyeoperations/puppetproduction+1 -3
ci: avoid hardcoded IP in Hiera, lookup contint.wikimedia.orgoperations/puppetproduction+6 -4
ci/zuul: use localhost as gearman serveroperations/puppetproduction+2 -2
zuul: add DNS lookup for gearman server IPoperations/puppetproduction+2 -0
jjb: switch contint2002 to contint1002integration/configmaster+10 -10
ci: switch source and destination server for data rsyncoperations/puppetproduction+2 -2
ci: switch gearman_server IP from contint2002 to contint1002operations/puppetproduction+2 -2
ci: switch contint manager_host from 2002 to 1002operations/puppetproduction+2 -2
switch contint.wikimedia.org from contint2002 to contint1002operations/dnsmaster+1 -1
ci: disable zuul merger on contint2002 for migrationoperations/puppetproduction+1 -0
contint: set bullseye docker version just for host contint1002operations/puppetproduction+1 -0
contint: disable zuul merger on contint1002, preparing for reimageoperations/puppetproduction+1 -0
zuul: require python2.7operations/puppetproduction+1 -1
contint: use the same PHP packages on contint before and after distro upgradeoperations/puppetproduction+3 -1
Use same php version for doc and integration websitesoperations/puppetproduction+38 -26
Show related patches Customize query in gerrit

Related Objects
Search...

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Apr 16 2024, 7:47 PM

Mentioned in SAL (#wikimedia-operations) [2024-04-16T19:47:44Z] <hashar@deploy1002> Started deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517

Stashbot added a comment.Apr 16 2024, 7:47 PM

Mentioned in SAL (#wikimedia-operations) [2024-04-16T19:47:52Z] <hashar@deploy1002> Finished deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517 (duration: 00m 08s)

Dzahn updated the task description. (Show Details)Apr 16 2024, 7:58 PM
gerritbot added a comment.Apr 16 2024, 8:06 PM

Change #1020344 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] contint: set new default docker version for bullseye

https://gerrit.wikimedia.org/r/1020344

Dzahn updated the task description. (Show Details)Apr 16 2024, 8:24 PM
Dzahn added a comment.Apr 16 2024, 8:28 PM

Screenshot from 2024-04-16 13-28-07.png (647×1 px, 41 KB)

Stashbot added a comment.Apr 16 2024, 8:30 PM

Mentioned in SAL (#wikimedia-operations) [2024-04-16T20:30:21Z] <mutante> CI - jenkins and zuul-merger are re-enabled on contint1002 after distro upgrade to bullseye - T334517

Dzahn added a subtask: T157818: zuul-merger fails when repository names overlaps.Apr 17 2024, 9:08 PM
Dzahn updated the task description. (Show Details)Apr 17 2024, 9:24 PM
Dzahn added a comment.Apr 17 2024, 10:31 PM

kind of weird: contint1002 and contint2002 both have rsyncd running and rsync snippets once created by puppet but I can't find the code in puppet and there are references to old server contint2001 that are nowhere in the repo either.

it's almost like puppetized rsync existed and was removed and just the remnants weren't deleted by puppet.

Dzahn added a comment.Apr 17 2024, 10:39 PM

see https://gerrit.wikimedia.org/r/c/operations/puppet/+/587491/5/modules/profile/manifests/ci/data_rsync.pp and https://gerrit.wikimedia.org/r/c/operations/puppet/+/606394/2/hieradata/role/common/ci/master.yaml

It's profile::manifests::ci::data_rsync but values are profile::ci::migration::* in role/common/ci.yaml.

Dzahn added a comment.Apr 17 2024, 10:49 PM

data pathes and sizes:

existing primary server:

root@contint2002:/# du -hs /var/lib/jenkins
2.2G	/var/lib/jenkins
root@contint2002:/# du -hs /var/lib/zuul/
6.0M	/var/lib/zuul/
root@contint2002:/# du -hs /srv/jenkins
291G	/srv/jenkins

reimaged server:

root@contint1002:/# du -hs /var/lib/jenkins
3.0M	/var/lib/jenkins
root@contint1002:/# du -hs /var/lib/zuul/
44K	/var/lib/zuul/
root@contint1002:/# du -hs /srv/jenkins
4.0K	/srv/jenkins
Stashbot added a comment.Apr 17 2024, 11:14 PM

Mentioned in SAL (#wikimedia-operations) [2024-04-17T23:14:01Z] <mutante> rsyncing jenkins data from contint2002 to contint1002, pre-sync in preparation for migration next week - /srv/jenkins (291G) and much smaller zuul and jenkins data dirs T334517

gerritbot added a comment.Apr 17 2024, 11:26 PM

Change #1020950 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: disable zuul merger on contint2002 for migration

https://gerrit.wikimedia.org/r/1020950

Dzahn updated the task description. (Show Details)Apr 17 2024, 11:29 PM
gerritbot added a comment.Apr 17 2024, 11:31 PM

Change #1020951 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/dns@master] switch contint.wikimedia.org from contint2002 to contint1002

https://gerrit.wikimedia.org/r/1020951

gerritbot added a comment.Apr 17 2024, 11:38 PM

Change #1020954 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: switch contint manager_host from 2002 to 1002

https://gerrit.wikimedia.org/r/1020954

Dzahn added a comment.Apr 17 2024, 11:42 PM

bugs in runbook:

  • there is no contint.discovery.wmnet name - it's contint.wikimedia.org
  • there is also a "gearman_server" setting which is an IP address hardcoded in Hiera and it's not mentioned in the runbook (profile::zuul::merger::conf)
  • doesn't mention rsync source and dest host settings in Hiera, which need to be switched with the failover
  • at the end, delete host-name based special settings like the docker version that are now globally the same again (https://gerrit.wikimedia.org/r/c/operations/puppet/+/1020316)
gerritbot added a comment.Apr 17 2024, 11:47 PM

Change #1020955 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: switch gearman_server IP from contint2002 to contint1002

https://gerrit.wikimedia.org/r/1020955

Dzahn updated the task description. (Show Details)Apr 17 2024, 11:50 PM

@hashar topic branch: https://gerrit.wikimedia.org/r/q/topic:%22contint-bullseye%22

gerritbot added a comment.Apr 17 2024, 11:52 PM

Change #1020957 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: switch source and destination server for data rsync

https://gerrit.wikimedia.org/r/1020957

gerritbot added a comment.Apr 18 2024, 12:02 AM

Change #1020958 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: avoid hardcoded IP in Hiera, lookup contint.wikimedia.org

https://gerrit.wikimedia.org/r/1020958

Dzahn added a comment.Apr 18 2024, 12:09 AM
  • confirmed rsync commands are working and executed them as noted in the runbook above - /srv/jenkins , /var/lib/zuul and /var/lib/jenkins have been pre-synced to contint1002
LSobanski moved this task from Work in Progress (Tracking tasks) to Work in Progress on the collaboration-services board.Apr 19 2024, 3:13 PM
Dzahn added a comment.May 6 2024, 8:41 PM

@hashar This happened today:

19:50 < Reedy> !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1028587
20:08 <+wikibugs> (Merged) jenkins-bot: jjb: Add some ruby3.1 jobs [integration/config] - https://gerrit.wikimedia.org/r/1028593 (owner: Reedy)
20:08 <+wikibugs> (PS1) Reedy: jjb: Add ruby3.1 mediawiki-ruby-api job [integration/config] - https://gerrit.wikimedia.org/r/1028594

20:29 < Reedy> !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1028596

19:50 < Reedy> !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1028587

Do we have to think of that when switching the server over? Asking because the "on contint primary" caught my eye.

hashar added a comment.May 7 2024, 6:33 AM

on contint primary

That is from the deployment script https://gerrit.wikimedia.org/g/integration/config/+/refs/heads/master/fab which refers to the contint.wikimedia.org hostname. That is thus switching once the DNS switch has settled :)

Dzahn awarded a token.May 7 2024, 5:29 PM
Jelto mentioned this in T364427: SystemdUnitFailed (contint1003).May 8 2024, 9:46 AM
Stashbot added a comment.May 13 2024, 2:09 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-13T14:09:04Z] <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on contint2002.wikimedia.org with reason: T334517

Stashbot added a comment.May 13 2024, 2:09 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-13T14:09:19Z] <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on contint2002.wikimedia.org with reason: T334517

Stashbot added a comment.May 13 2024, 2:09 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-13T14:09:40Z] <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 1:00:00 on contint1002.wikimedia.org with reason: T334517

Stashbot added a comment.May 13 2024, 2:09 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-13T14:09:55Z] <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on contint1002.wikimedia.org with reason: T334517

gerritbot added a comment.May 13 2024, 2:11 PM

Change #1020950 merged by Dzahn:

[operations/puppet@production] ci: disable zuul merger on contint2002 for migration

https://gerrit.wikimedia.org/r/1020950

Dzahn updated the task description. (Show Details)May 13 2024, 2:12 PM
Stashbot added a comment.May 13 2024, 2:15 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-13T14:15:11Z] <mutante> CI - migration in progress - stopping jenkins and zuul (T334517)

gerritbot added a comment.May 13 2024, 2:18 PM

Change #1020951 merged by Dzahn:

[operations/dns@master] switch contint.wikimedia.org from contint2002 to contint1002

https://gerrit.wikimedia.org/r/1020951

jnuche updated the task description. (Show Details)May 13 2024, 2:18 PM
Dzahn updated the task description. (Show Details)May 13 2024, 2:19 PM
gerritbot added a comment.May 13 2024, 2:22 PM

Change #1020954 merged by Dzahn:

[operations/puppet@production] ci: switch contint manager_host from 2002 to 1002

https://gerrit.wikimedia.org/r/1020954

gerritbot added a comment.May 13 2024, 2:23 PM

Change #1020955 merged by Dzahn:

[operations/puppet@production] ci: switch gearman_server IP from contint2002 to contint1002

https://gerrit.wikimedia.org/r/1020955

jnuche updated the task description. (Show Details)May 13 2024, 2:28 PM
Dzahn updated the task description. (Show Details)May 13 2024, 2:28 PM
gerritbot added a comment.May 13 2024, 2:32 PM

Change #1020957 merged by Dzahn:

[operations/puppet@production] ci: switch source and destination server for data rsync

https://gerrit.wikimedia.org/r/1020957

Dzahn updated the task description. (Show Details)May 13 2024, 2:33 PM
Stashbot added a comment.May 13 2024, 2:34 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-13T14:34:17Z] <mutante> CI - switch over to other contint server finished - T334517

gerritbot added a comment.May 13 2024, 2:42 PM

Change #1030991 had a related patch set uploaded (by Hashar; author: Hashar):

[integration/config@master] jjb: switch contint2002 to contint1002

https://gerrit.wikimedia.org/r/1030991

gerritbot added a comment.May 13 2024, 2:48 PM

Change #1030991 merged by jenkins-bot:

[integration/config@master] jjb: switch contint2002 to contint1002

https://gerrit.wikimedia.org/r/1030991

hashar awarded a token.May 13 2024, 2:53 PM
Stashbot added a comment.May 13 2024, 3:18 PM

Mentioned in SAL (#wikimedia-releng) [2024-05-13T15:18:42Z] <hashar> deployment-prep: deleted security rule for 208.80.154.17 ssh and port 2 (sic) and allow 208.80.154.132 / contint1002 port 22 instead # T334517

gerritbot added a comment.Wed, May 15, 6:22 PM

Change #1020958 merged by Dzahn:

[operations/puppet@production] ci: avoid hardcoded IP in Hiera, lookup contint.wikimedia.org

https://gerrit.wikimedia.org/r/1020958

gerritbot added a comment.Wed, May 15, 6:39 PM

Change #1032010 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci/zuul: use localhost as gearman server

https://gerrit.wikimedia.org/r/1032010

gerritbot added a comment.Wed, May 15, 6:39 PM

Change #1032010 merged by Dzahn:

[operations/puppet@production] ci/zuul: use localhost as gearman server

https://gerrit.wikimedia.org/r/1032010

gerritbot added a comment.Wed, May 15, 6:50 PM

Change #1032013 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] zuul: add DNS lookup for gearman server IP

https://gerrit.wikimedia.org/r/1032013

gerritbot added a comment.Wed, May 15, 6:52 PM

Change #1032013 merged by Dzahn:

[operations/puppet@production] zuul: add DNS lookup for gearman server IP

https://gerrit.wikimedia.org/r/1032013

gerritbot added a comment.Wed, May 15, 7:17 PM

Change #1032023 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] ci: set puppet7 on role level

https://gerrit.wikimedia.org/r/1032023

gerritbot added a comment.Wed, May 15, 7:59 PM

Change #1020344 merged by Dzahn:

[operations/puppet@production] contint: set new default docker version for bullseye

https://gerrit.wikimedia.org/r/1020344

Stashbot added a comment.Thu, May 16, 2:43 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T14:43:02Z] <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 3:00:00 on contint2002.wikimedia.org with reason: T334517

Stashbot added a comment.Thu, May 16, 2:43 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T14:43:18Z] <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on contint2002.wikimedia.org with reason: T334517

ops-monitoring-bot added a comment.Thu, May 16, 3:03 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host contint2002.wikimedia.org with OS bullseye

gerritbot added a comment.Thu, May 16, 3:13 PM

Change #1032023 merged by Dzahn:

[operations/puppet@production] ci: set puppet7 at role level

https://gerrit.wikimedia.org/r/1032023

ops-monitoring-bot added a comment.Thu, May 16, 3:24 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host contint2002.wikimedia.org with OS bullseye executed with errors:

  • contint2002 (FAIL)
    • Downtimed on Icinga/Alertmanager
    • Disabled Puppet
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" contint2002.wikimedia.org to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, May 16, 3:25 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host contint2002.wikimedia.org with OS bullseye

Maintenance_bot removed a project: Patch-For-Review.Thu, May 16, 3:32 PM
ops-monitoring-bot added a comment.Thu, May 16, 4:40 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host contint2002.wikimedia.org with OS bullseye executed with errors:

  • contint2002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" contint2002.wikimedia.org to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, May 16, 4:41 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host contint2002.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Thu, May 16, 4:57 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host contint2002.wikimedia.org with OS bullseye executed with errors:

  • contint2002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" contint2002.wikimedia.org to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, May 16, 4:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host contint2002.wikimedia.org with OS buster

ops-monitoring-bot added a comment.Thu, May 16, 6:04 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin2002 for host contint2002.wikimedia.org with OS buster executed with errors:

  • contint2002 (FAIL)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • The reimage failed, see the cookbook logs for the details,You can also try typing "install-console" contint2002.wikimedia.org to get a root shellbut depending on the failure this may not work.
ops-monitoring-bot added a comment.Thu, May 16, 6:17 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host contint2002.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Thu, May 16, 6:58 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin2002 for host contint2002.wikimedia.org with OS buster

ops-monitoring-bot added a comment.Thu, May 16, 7:55 PM

Cookbook cookbooks.sre.hosts.reimage was started by dzahn@cumin1002 for host contint2002.wikimedia.org with OS bullseye

ops-monitoring-bot added a comment.Thu, May 16, 8:33 PM

Cookbook cookbooks.sre.hosts.reimage started by dzahn@cumin1002 for host contint2002.wikimedia.org with OS bullseye completed:

  • contint2002 (PASS)
    • Removed from Puppet and PuppetDB if present and deleted any certificates
    • Removed from Debmonitor if present
    • Forced PXE for next reboot
    • Host rebooted via IPMI
    • Host up (Debian installer)
    • Add puppet_version metadata to Debian installer
    • Checked BIOS boot parameters are back to normal
    • Host up (new fresh bullseye OS)
    • Generated Puppet certificate
    • Signed new Puppet certificate
    • Run Puppet in NOOP mode to populate exported resources in PuppetDB
    • Found Nagios_host resource for this host in PuppetDB
    • Downtimed the new host on Icinga/Alertmanager
    • First Puppet run completed and logged in /var/log/spicerack/sre/hosts/reimage/202405162014_dzahn_464740_contint2002.out
    • configmaster.wikimedia.org updated with the host new SSH public key for wmf-update-known-hosts-production
    • Rebooted
    • Automatic Puppet run was successful
    • Forced a re-check of all Icinga services for the host
    • Icinga status is optimal
    • Icinga downtime removed
    • Updated Netbox data from PuppetDB
    • Cleared switch DHCP cache and MAC table for the host IP and MAC (EVPN Switch)
Dzahn updated the task description. (Show Details)Thu, May 16, 8:35 PM
Dzahn added a comment.Thu, May 16, 8:37 PM

After a couple problems with reimaging (networking config via DHCP in Debian installer failing due to networking changes, RAID setup failure due to changing device names from attached cdrom drive, cookbook locks override, apache module race condition on first puppet run), now contint2002 has been reimaged with bullseye.

jenkins and zuul services are masked as they should be.

Dzahn closed this task as Resolved.Thu, May 16, 8:38 PM
hashar reopened this task as Open.Wed, May 22, 7:33 AM

Congratulations on having reimaged contint2002! It is missing steps though:

  • attached as a Jenkins agent to the Jenkins controller which runs on the other host
  • running the secondary zuul-merger daemon (and its companion git-daemon)

And we can also get rid of the Ganeti VM contint1003.eqiad.wmnet which was created for testing Zuul under Bullseye ( T361224 ).

Stashbot added a comment.Wed, May 22, 7:35 AM

Mentioned in SAL (#wikimedia-releng) [2024-05-22T07:35:42Z] <hashar> Jenkins: added back contint2002 as an Agent. The host got reimaged from Buster to Bullseye # T334517

Dzahn added a comment.Wed, May 22, 2:52 PM
In T334517#9819898, @hashar wrote:

And we can also get rid of the Ganeti VM contint1003.eqiad.wmnet

I wanted to ask you about this anyways. Would have just used another ticket. I'll get right on that.

gerritbot added a comment.Wed, May 22, 3:02 PM

Change #1034954 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] site: remove contint1003 - former testing machine

https://gerrit.wikimedia.org/r/1034954

gerritbot added a project: Patch-For-Review.Wed, May 22, 3:02 PM
gerritbot added a comment.Wed, May 22, 3:04 PM

Change #1034955 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] role: delete ci_test role, not used anymore

https://gerrit.wikimedia.org/r/1034955

gerritbot added a comment.Wed, May 22, 3:21 PM

Change #1034954 merged by Dzahn:

[operations/puppet@production] site: remove contint1003 - former testing machine

https://gerrit.wikimedia.org/r/1034954

gerritbot added a comment.Wed, May 22, 3:22 PM

Change #1034965 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] hieradata: delete host contint1003

https://gerrit.wikimedia.org/r/1034965

gerritbot added a comment.Wed, May 22, 3:23 PM

Change #1034965 merged by Dzahn:

[operations/puppet@production] hieradata: delete host contint1003

https://gerrit.wikimedia.org/r/1034965

Dzahn added a comment.Wed, May 22, 4:43 PM

@hashar contint1003 is removed. What's next? Do we need a Hiera change to unmask any services?

gerritbot added a comment.Thu, May 23, 2:38 PM

Change #1034955 merged by Dzahn:

[operations/puppet@production] role: delete ci_test role, not used anymore

https://gerrit.wikimedia.org/r/1034955

Maintenance_bot removed a project: Patch-For-Review.Thu, May 23, 3:32 PM
gerritbot added a comment.Tue, May 28, 10:18 PM

Change #1036762 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] contint: enable zuul-merger daemon on contint2002

https://gerrit.wikimedia.org/r/1036762

gerritbot added a project: Patch-For-Review.Tue, May 28, 10:18 PM
Dzahn added a comment.Tue, May 28, 10:29 PM
In T334517#9819898, @hashar wrote:

Congratulations on having reimaged contint2002! It is missing steps though:

  • attached as a Jenkins agent to the Jenkins controller which runs on the other host

How do you do that? Is this just an action in the integration.wikimedia.org web UI?

  • running the secondary zuul-merger daemon (and its companion git-daemon)

git-daemon is already running.

Active: active (running) since Thu 2024-05-16 20:30:55 UTC; 1 weeks 5 days ago

zuul-merger is masked and I uploaded https://gerrit.wikimedia.org/r/c/operations/puppet/+/1036762 to unmask it.

And we can also get rid of the Ganeti VM contint1003.eqiad.wmnet which was created for testing Zuul under Bullseye ( T361224 ).

done!

Dzahn added a comment.Tue, May 28, 10:32 PM

@hashar I see jobs at https://integration.wikimedia.org/ci/computer/contint2002/

Doesn't this mean the " attached as a Jenkins agent to the Jenkins controller" is done?

hashar added a comment.Wed, May 29, 12:36 PM
In T334517#9840346, @Dzahn wrote:

@hashar I see jobs at https://integration.wikimedia.org/ci/computer/contint2002/

Doesn't this mean the " attached as a Jenkins agent to the Jenkins controller" is done?

YES, see the comment I have made above when I have added it back:

In T334517#9819919, @Stashbot wrote:

Mentioned in SAL (#wikimedia-releng) [2024-05-22T07:35:42Z] <hashar> Jenkins: added back contint2002 as an Agent. The host got reimaged from Buster to Bullseye # T334517

gerritbot added a comment.Wed, May 29, 12:39 PM

Change #1036762 merged by Jelto:

[operations/puppet@production] contint: enable zuul-merger daemon on contint2002

https://gerrit.wikimedia.org/r/1036762

hashar closed this task as Resolved.Wed, May 29, 12:45 PM

zuul-merger is now running on contint2002.wikimedia.org

That concludes the OS upgrade to Bullseye for those two hosts.

Maintenance_bot removed a project: Patch-For-Review.Wed, May 29, 1:31 PM
Dzahn added a comment.Wed, May 29, 4:36 PM

:) cool, thanks for this and the explanation on IRC

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL