| 
      
        2024-11-21
      
      §
     | 
  
    
  | 10:08 | 
  <arnaudb@cumin1002> | 
  dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P71111 and previous config saved to /var/cache/conftool/dbconfig/20241121-100821-arnaudb.json | 
  [production] | 
            
  | 10:01 | 
  <dcausse@deploy2002> | 
  helmfile [codfw] DONE helmfile.d/services/eventgate-main: sync | 
  [production] | 
            
  | 10:01 | 
  <dcausse@deploy2002> | 
  helmfile [codfw] START helmfile.d/services/eventgate-main: sync | 
  [production] | 
            
  | 09:59 | 
  <dcausse> | 
  restarting eventgate-main@codfw | 
  [production] | 
            
  | 09:53 | 
  <arnaudb@cumin1002> | 
  dbctl commit (dc=all): 'Repooling after maintenance db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71110 and previous config saved to /var/cache/conftool/dbconfig/20241121-095313-arnaudb.json | 
  [production] | 
            
  | 09:51 | 
  <arnaudb@cumin1002> | 
  dbctl commit (dc=all): 'Depooling db2155 (T367781)', diff saved to https://phabricator.wikimedia.org/P71109 and previous config saved to /var/cache/conftool/dbconfig/20241121-095102-arnaudb.json | 
  [production] | 
            
  | 09:50 | 
  <arnaudb@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance | 
  [production] | 
            
  | 09:50 | 
  <arnaudb@cumin1002> | 
  START - Cookbook sre.hosts.downtime for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance | 
  [production] | 
            
  | 09:50 | 
  <arnaudb@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance | 
  [production] | 
            
  | 09:50 | 
  <arnaudb@cumin1002> | 
  START - Cookbook sre.hosts.downtime for 4:00:00 on db2155.codfw.wmnet with reason: Maintenance | 
  [production] | 
            
  | 09:35 | 
  <moritzm> | 
  installing nghttp2 security updates | 
  [production] | 
            
  | 09:18 | 
  <arnaudb@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1246.eqiad.wmnet with OS bookworm | 
  [production] | 
            
  | 09:17 | 
  <aklapper@deploy2002> | 
  rebuilt and synchronized wikiversions files: group2 to 1.44.0-wmf.4  refs T375663 | 
  [production] | 
            
  | 09:07 | 
  <moritzm> | 
  installing exim4 security updates | 
  [production] | 
            
  | 09:03 | 
  <arnaudb@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage | 
  [production] | 
            
  | 09:00 | 
  <arnaudb@cumin1002> | 
  START - Cookbook sre.hosts.downtime for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage | 
  [production] | 
            
  | 08:45 | 
  <arnaudb@cumin1002> | 
  START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm | 
  [production] | 
            
  | 08:21 | 
  <kartik@deploy2002> | 
  Finished scap sync-world: Backport for [[gerrit:1093733|Enable the Contribute menu in 4th group of Wikis (T375303)]] (duration: 14m 05s) | 
  [production] | 
            
  | 08:14 | 
  <kartik@deploy2002> | 
  kartik: Continuing with sync | 
  [production] | 
            
  | 08:09 | 
  <kartik@deploy2002> | 
  kartik: Backport for [[gerrit:1093733|Enable the Contribute menu in 4th group of Wikis (T375303)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) | 
  [production] | 
            
  | 08:06 | 
  <kartik@deploy2002> | 
  Started scap sync-world: Backport for [[gerrit:1093733|Enable the Contribute menu in 4th group of Wikis (T375303)]] | 
  [production] | 
            
  | 07:48 | 
  <moritzm> | 
  removing ganeti1017 from active Ganeti nodes T378921 | 
  [production] | 
            
  | 05:51 | 
  <aikochou@deploy2002> | 
  helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . | 
  [production] | 
            
  | 02:30 | 
  <brett> | 
  Import libvmod-re2_2.0.0-2~bpo11u1 into varnish-staging apt component | 
  [production] | 
            
  | 00:45 | 
  <urandom> | 
  decommissioning Cassandra/restbase2021-{a,b,c} — T380236 | 
  [production] | 
            
  | 00:42 | 
  <eevans@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase2023.codfw.wmnet with reason: Decommissioning — T380236 | 
  [production] | 
            
  | 00:42 | 
  <eevans@cumin1002> | 
  START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase2023.codfw.wmnet with reason: Decommissioning — T380236 | 
  [production] | 
            
  | 00:42 | 
  <eevans@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase2022.codfw.wmnet with reason: Decommissioning — T380236 | 
  [production] | 
            
  | 00:42 | 
  <eevans@cumin1002> | 
  START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase2022.codfw.wmnet with reason: Decommissioning — T380236 | 
  [production] | 
            
  | 00:42 | 
  <eevans@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on restbase2021.codfw.wmnet with reason: Decommissioning — T380236 | 
  [production] | 
            
  | 00:42 | 
  <eevans@cumin1002> | 
  START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on restbase2021.codfw.wmnet with reason: Decommissioning — T380236 | 
  [production] | 
            
  | 00:40 | 
  <eevans@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for restbase2038.codfw.wmnet | 
  [production] | 
            
  | 00:40 | 
  <eevans@cumin1002> | 
  START - Cookbook sre.hosts.remove-downtime for restbase2038.codfw.wmnet | 
  [production] | 
            
  | 00:40 | 
  <eevans@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for restbase2037.codfw.wmnet | 
  [production] | 
            
  | 00:40 | 
  <eevans@cumin1002> | 
  START - Cookbook sre.hosts.remove-downtime for restbase2037.codfw.wmnet | 
  [production] | 
            
  | 00:40 | 
  <eevans@cumin1002> | 
  END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for restbase2036.codfw.wmnet | 
  [production] | 
            
  | 00:40 | 
  <eevans@cumin1002> | 
  START - Cookbook sre.hosts.remove-downtime for restbase2036.codfw.wmnet | 
  [production] | 
            
  | 00:15 | 
  <urbanecm> | 
  [urbanecm@deploy2002 ~]$ mwscript-k8s -- extensions/GrowthExperiments/maintenance/revalidateLinkRecommendations.php --wiki=azwiki --all --verbose # T380329 | 
  [production] | 
            
  
    | 
      
        2024-11-20
      
      §
     | 
  
    
  | 23:54 | 
  <Reedy> | 
  Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1093462 T380434 | 
  [releng] | 
            
  | 23:45 | 
  <Reedy> | 
  Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1093459 T380434 | 
  [releng] | 
            
  | 23:22 | 
  <cjming> | 
  end of UTC late backport window | 
  [production] | 
            
  | 23:20 | 
  <eileen> | 
  civicrm upgraded from 7c940d6f to 3311520a | 
  [production] | 
            
  | 23:17 | 
  <cjming@deploy2002> | 
  Finished scap sync-world: Backport for [[gerrit:1093408|Temporarily disable dark mode for anonymous users (T379765)]] (duration: 13m 06s) | 
  [production] | 
            
  | 23:10 | 
  <cjming@deploy2002> | 
  jdlrobson, cjming: Continuing with sync | 
  [production] | 
            
  | 23:08 | 
  <cjming@deploy2002> | 
  jdlrobson, cjming: Backport for [[gerrit:1093408|Temporarily disable dark mode for anonymous users (T379765)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) | 
  [production] | 
            
  | 23:04 | 
  <cjming@deploy2002> | 
  Started scap sync-world: Backport for [[gerrit:1093408|Temporarily disable dark mode for anonymous users (T379765)]] | 
  [production] | 
            
  | 23:03 | 
  <cjming@deploy2002> | 
  Finished scap sync-world: Backport for [[gerrit:1093328|knwiki: update portal namespace (T380366)]] (duration: 12m 17s) | 
  [production] | 
            
  | 22:56 | 
  <cjming@deploy2002> | 
  cjming, anzx: Continuing with sync | 
  [production] | 
            
  | 22:55 | 
  <cjming@deploy2002> | 
  cjming, anzx: Backport for [[gerrit:1093328|knwiki: update portal namespace (T380366)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) | 
  [production] | 
            
  | 22:52 | 
  <brett> | 
  Import libvmod-querysort 0.4-3 into varnish-staging apt component | 
  [production] |