3601-3650 of 10000 results (21ms)
2025-03-24 §
08:55 <elukey@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1119.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
08:55 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host elastic1119.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
08:54 <tgr@deploy1003> tgr: Backport for [[gerrit:1130320|authmanager: Use an URL parameter to keep track of returns (T388250)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:49 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host elastic1119.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
08:48 <tgr@deploy1003> Started scap sync-world: Backport for [[gerrit:1130320|authmanager: Use an URL parameter to keep track of returns (T388250)]] [production]
08:48 <dcausse@deploy1003> helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
08:48 <dcausse@deploy1003> helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [production]
08:47 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-serve2004.codfw.wmnet with reason: host reimage [production]
08:45 <elukey@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on ml-serve2004.codfw.wmnet with reason: host reimage [production]
08:45 <tgr@deploy1003> Finished scap sync-world: Backport for [[gerrit:1130328|Preserve 'useformat' param when accessing Special:ChangePassword]] (duration: 33m 43s) [production]
08:43 <marostegui@cumin1002> dbctl commit (dc=all): 'db2204 (re)pooling @ 50%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P74313 and previous config saved to /var/cache/conftool/dbconfig/20250324-084309-root.json [production]
08:42 <dcausse@deploy1003> helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
08:42 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host elastic1119.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
08:41 <dcausse@deploy1003> helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply [production]
08:39 <elukey@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host elastic1119.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
08:38 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host elastic1119.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
08:36 <dcausse@deploy1003> helmfile [staging] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
08:36 <dcausse@deploy1003> helmfile [staging] START helmfile.d/services/cirrus-streaming-updater: apply [production]
08:35 <tgr@deploy1003> tgr: Continuing with sync [production]
08:30 <tgr@deploy1003> tgr: Backport for [[gerrit:1130328|Preserve 'useformat' param when accessing Special:ChangePassword]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
08:28 <marostegui@cumin1002> dbctl commit (dc=all): 'db2204 (re)pooling @ 25%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P74312 and previous config saved to /var/cache/conftool/dbconfig/20250324-082804-root.json [production]
08:27 <elukey@cumin1002> START - Cookbook sre.hosts.reimage for host ml-serve2004.codfw.wmnet with OS bookworm [production]
08:12 <marostegui@cumin1002> dbctl commit (dc=all): 'db2204 (re)pooling @ 10%: Repooling after rebuild index', diff saved to https://phabricator.wikimedia.org/P74311 and previous config saved to /var/cache/conftool/dbconfig/20250324-081258-root.json [production]
08:11 <tgr@deploy1003> Started scap sync-world: Backport for [[gerrit:1130328|Preserve 'useformat' param when accessing Special:ChangePassword]] [production]
07:28 <moritzm> rebalance ganeti eqiad/D following reimages T382507 [production]
07:16 <marostegui@cumin1002> END (PASS) - Cookbook sre.mysql.depool (exit_code=0) db2214 - Depool db2169.codfw.wmnet to then clone it to db2214.codfw.wmnet - marostegui@cumin1002 [production]
07:16 <marostegui@cumin1002> START - Cookbook sre.mysql.depool db2214 - Depool db2169.codfw.wmnet to then clone it to db2214.codfw.wmnet - marostegui@cumin1002 [production]
07:16 <marostegui@cumin1002> START - Cookbook sre.mysql.clone of db2169.codfw.wmnet onto db2214.codfw.wmnet [production]
07:08 <marostegui@cumin1002> END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2214.codfw.wmnet [production]
07:04 <marostegui@cumin1002> START - Cookbook sre.mysql.upgrade for db2214.codfw.wmnet [production]
07:02 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2214 T389382', diff saved to https://phabricator.wikimedia.org/P74309 and previous config saved to /var/cache/conftool/dbconfig/20250324-070245-marostegui.json [production]
07:01 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db2229 to s6 primary T389382', diff saved to https://phabricator.wikimedia.org/P74308 and previous config saved to /var/cache/conftool/dbconfig/20250324-070147-marostegui.json [production]
07:01 <marostegui> Starting s6 codfw failover from db2214 to db2229 - T389382 [production]
06:53 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s6 T389382 [production]
06:52 <marostegui@cumin1002> dbctl commit (dc=all): 'Set db2229 with weight 0 T389382', diff saved to https://phabricator.wikimedia.org/P74307 and previous config saved to /var/cache/conftool/dbconfig/20250324-065236-marostegui.json [production]
06:32 <root@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2204.codfw.wmnet with reason: Index rebuild [production]
06:30 <marostegui@cumin1002> END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2204.codfw.wmnet [production]
06:26 <marostegui@cumin1002> START - Cookbook sre.mysql.upgrade for db2204.codfw.wmnet [production]
06:23 <marostegui@cumin1002> dbctl commit (dc=all): 'Depool db2204 T389376', diff saved to https://phabricator.wikimedia.org/P74306 and previous config saved to /var/cache/conftool/dbconfig/20250324-062338-marostegui.json [production]
06:22 <marostegui@cumin1002> dbctl commit (dc=all): 'Promote db2207 to s2 primary T389376', diff saved to https://phabricator.wikimedia.org/P74305 and previous config saved to /var/cache/conftool/dbconfig/20250324-062223-marostegui.json [production]
06:22 <marostegui> Starting s2 codfw failover from db2204 to db2207 - T389376 [production]
06:18 <marostegui@cumin1002> dbctl commit (dc=all): 'Set db2207 with weight 0 T389376', diff saved to https://phabricator.wikimedia.org/P74304 and previous config saved to /var/cache/conftool/dbconfig/20250324-061812-marostegui.json [production]
06:18 <marostegui@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s2 T389376 [production]
2025-03-23 §
18:41 <James_F> Zuul: Add 0xDeadbeef to CI allowlist [releng]
18:34 <James_F> Zuul: [operations/debs/bdsync] Mark as archived, for T377882 [releng]
18:31 <James_F> Zuul: [mediawiki/extensions/CheckUser] Add GrowthExperiments dependency, for T386435 [releng]
18:29 <James_F> Zuul: [mediawiki/extensions/CategoryWatch] Add Echo CI dependency [releng]
2025-03-22 §
07:20 <vgutierrez@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on cp4047.ulsfo.wmnet with reason: HW errors [production]
07:15 <vgutierrez[off]> restart purged on cp7001 - T389707 [production]
04:00 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-44 [tools]