| 2024-11-21
      
      § | 
    
  | 09:51 | <arnaudb@cumin1002> | dbctl commit (dc=all): 'Depooling db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71109 and previous config saved to /var/cache/conftool/dbconfig/20241121-095102-arnaudb.json | [production] | 
            
  | 09:50 | <arnaudb@cumin1002> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance | [production] | 
            
  | 09:50 | <arnaudb@cumin1002> | START - Cookbook sre.hosts.downtime for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance | [production] | 
            
  | 09:50 | <arnaudb@cumin1002> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance | [production] | 
            
  | 09:50 | <arnaudb@cumin1002> | START - Cookbook sre.hosts.downtime for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance | [production] | 
            
  | 09:35 | <moritzm> | installing nghttp2 security updates | [production] | 
            
  | 09:18 | <arnaudb@cumin1002> | END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1246.eqiad.wmnet with OS bookworm | [production] | 
            
  | 09:17 | <aklapper@deploy2002> | rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.4  refs T375663 | [production] | 
            
  | 09:07 | <moritzm> | installing exim4 security updates | [production] | 
            
  | 09:03 | <arnaudb@cumin1002> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage | [production] | 
            
  | 09:00 | <arnaudb@cumin1002> | START - Cookbook sre.hosts.downtime for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage | [production] | 
            
  | 08:45 | <arnaudb@cumin1002> | START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm | [production] | 
            
  | 08:21 | <kartik@deploy2002> | Finished scap sync-world: Backport for [[gerrit:1093733|Enable the Contribute menu in 4th group of Wikis (T375303)]] (duration: 14m 05s) | [production] | 
            
  | 08:14 | <kartik@deploy2002> | kartik: Continuing with sync | [production] | 
            
  | 08:09 | <kartik@deploy2002> | kartik: Backport for [[gerrit:1093733|Enable the Contribute menu in 4th group of Wikis (T375303)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) | [production] | 
            
  | 08:06 | <kartik@deploy2002> | Started scap sync-world: Backport for [[gerrit:1093733|Enable the Contribute menu in 4th group of Wikis (T375303)]] | [production] | 
            
  | 07:48 | <moritzm> | removing ganeti1017 from active Ganeti nodes T378921 | [production] | 
            
  | 05:51 | <aikochou@deploy2002> | helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . | [production] | 
            
  | 02:30 | <brett> | Import libvmod-re2_2.0.0-2~bpo11u1 into varnish-staging apt component | [production] | 
            
  | 00:45 | <urandom> | decommissioning Cassandra/restbase2021-{a,b,c} — T380236 | [production] | 
            
  | 00:42 | <eevans@cumin1002> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase2023.codfw.wmnet with reason: Decommissioning — T380236 | [production] | 
            
  | 00:42 | <eevans@cumin1002> | START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase2023.codfw.wmnet with reason: Decommissioning — T380236 | [production] | 
            
  | 00:42 | <eevans@cumin1002> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase2022.codfw.wmnet with reason: Decommissioning — T380236 | [production] | 
            
  | 00:42 | <eevans@cumin1002> | START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase2022.codfw.wmnet with reason: Decommissioning — T380236 | [production] | 
            
  | 00:42 | <eevans@cumin1002> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase2021.codfw.wmnet with reason: Decommissioning — T380236 | [production] | 
            
  | 00:42 | <eevans@cumin1002> | START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase2021.codfw.wmnet with reason: Decommissioning — T380236 | [production] | 
            
  | 00:40 | <eevans@cumin1002> | END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for restbase2038.codfw.wmnet | [production] | 
            
  | 00:40 | <eevans@cumin1002> | START - Cookbook sre.hosts.remove-downtime for restbase2038.codfw.wmnet | [production] | 
            
  | 00:40 | <eevans@cumin1002> | END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for restbase2037.codfw.wmnet | [production] | 
            
  | 00:40 | <eevans@cumin1002> | START - Cookbook sre.hosts.remove-downtime for restbase2037.codfw.wmnet | [production] | 
            
  | 00:40 | <eevans@cumin1002> | END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for restbase2036.codfw.wmnet | [production] | 
            
  | 00:40 | <eevans@cumin1002> | START - Cookbook sre.hosts.remove-downtime for restbase2036.codfw.wmnet | [production] | 
            
  | 00:15 | <urbanecm> | [urbanecm@deploy2002 ~]$ mwscript-k8s -- extensions/GrowthExperiments/maintenance/revalidateLinkRecommendations.php --wiki=azwiki --all --verbose # T380329 | [production] | 
            
  
    | 2024-11-20
      
      § | 
    
  | 23:22 | <cjming> | end of UTC late backport window | [production] | 
            
  | 23:20 | <eileen> | civicrm upgraded from 7c940d6f to 3311520a | [production] | 
            
  | 23:17 | <cjming@deploy2002> | Finished scap sync-world: Backport for [[gerrit:1093408|Temporarily disable dark mode for anonymous users (T379765)]] (duration: 13m 06s) | [production] | 
            
  | 23:10 | <cjming@deploy2002> | jdlrobson, cjming: Continuing with sync | [production] | 
            
  | 23:08 | <cjming@deploy2002> | jdlrobson, cjming: Backport for [[gerrit:1093408|Temporarily disable dark mode for anonymous users (T379765)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) | [production] | 
            
  | 23:04 | <cjming@deploy2002> | Started scap sync-world: Backport for [[gerrit:1093408|Temporarily disable dark mode for anonymous users (T379765)]] | [production] | 
            
  | 23:03 | <cjming@deploy2002> | Finished scap sync-world: Backport for [[gerrit:1093328|knwiki: update portal namespace (T380366)]] (duration: 12m 17s) | [production] | 
            
  | 22:56 | <cjming@deploy2002> | cjming, anzx: Continuing with sync | [production] | 
            
  | 22:55 | <cjming@deploy2002> | cjming, anzx: Backport for [[gerrit:1093328|knwiki: update portal namespace (T380366)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) | [production] | 
            
  | 22:52 | <brett> | Import libvmod-querysort 0.4-3 into varnish-staging apt component | [production] | 
            
  | 22:51 | <cjming@deploy2002> | Started scap sync-world: Backport for [[gerrit:1093328|knwiki: update portal namespace (T380366)]] | [production] | 
            
  | 22:49 | <cjming@deploy2002> | Finished scap sync-world: Backport for [[gerrit:1093446|Revert "Add contact form for U4C"]] (duration: 14m 22s) | [production] | 
            
  | 22:49 | <jhathaway@cumin2002> | END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host thanos-be2005.codfw.wmnet with OS bullseye | [production] | 
            
  | 22:41 | <bking@deploy2002> | helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/blunderbuss: apply | [production] | 
            
  | 22:41 | <bking@deploy2002> | helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/blunderbuss: apply | [production] | 
            
  | 22:40 | <cjming@deploy2002> | trainbranchbot, cjming: Continuing with sync | [production] | 
            
  | 22:40 | <cjming@deploy2002> | trainbranchbot, cjming: Backport for [[gerrit:1093446|Revert "Add contact form for U4C"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) | [production] |