351-400 of 10000 results (26ms)
2025-08-29 §
01:00 <mwpresync@deploy1003> Started scap build-images: Publishing wmf/next image [production]
00:13 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db2149 (T402925)', diff saved to https://phabricator.wikimedia.org/P82072 and previous config saved to /var/cache/conftool/dbconfig/20250829-001328-ladsgroup.json [production]
00:13 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2149.codfw.wmnet with reason: Maintenance [production]
00:10 <wmbot~damian-scripts@tools-bastion-13> report deployed @ refs/tags/v1.1.1 [tools.cluebotng]
00:10 <wmbot~damian-scripts@tools-bastion-13> report deployed @ refs/heads/main [tools.cluebotng-staging]
00:01 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.osd.undrain_node [admin]
2025-08-28 §
23:38 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host maps2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
23:22 <krinkle@deploy1003> Finished scap sync-world: Backport for [[gerrit:1181310|Enable wmgUseMdotRouting in Beta Cluster for testwiki only (T401595)]] (duration: 10m 54s) [production]
23:20 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host maps2012.codfw.wmnet with OS bookworm [production]
23:16 <krinkle@deploy1003> krinkle: Continuing with sync [production]
23:16 <rzl@deploy1003> helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [production]
23:16 <rzl@deploy1003> helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [production]
23:14 <jclark@cumin1002> START - Cookbook sre.hosts.provision for host maps2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
23:13 <krinkle@deploy1003> krinkle: Backport for [[gerrit:1181310|Enable wmgUseMdotRouting in Beta Cluster for testwiki only (T401595)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
23:13 <rzl@deploy1003> helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [production]
23:12 <rzl@deploy1003> helmfile [codfw] START helmfile.d/services/rest-gateway: apply [production]
23:12 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance [production]
23:11 <krinkle@deploy1003> Started scap sync-world: Backport for [[gerrit:1181310|Enable wmgUseMdotRouting in Beta Cluster for testwiki only (T401595)]] [production]
23:08 <rzl@deploy1003> helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply [production]
23:07 <rzl@deploy1003> helmfile [eqiad] START helmfile.d/services/api-gateway: apply [production]
23:05 <rzl@deploy1003> helmfile [codfw] DONE helmfile.d/services/api-gateway: apply [production]
23:05 <rzl@deploy1003> helmfile [codfw] START helmfile.d/services/api-gateway: apply [production]
22:56 <rzl@deploy1003> helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [production]
22:55 <rzl@deploy1003> helmfile [staging] START helmfile.d/services/rest-gateway: apply [production]
22:36 <rzl@deploy1003> Finished scap sync-world: https://gerrit.wikimedia.org/r/1182945 (duration: 06m 50s) [production]
22:31 <rzl@deploy1003> Started scap sync-world: https://gerrit.wikimedia.org/r/1182945 [production]
22:18 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
22:17 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T402925)', diff saved to https://phabricator.wikimedia.org/P82071 and previous config saved to /var/cache/conftool/dbconfig/20250828-221753-ladsgroup.json [production]
22:11 <maryum> Security deploy for T403093 [production]
22:02 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P82070 and previous config saved to /var/cache/conftool/dbconfig/20250828-220245-ladsgroup.json [production]
22:02 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host maps2011.codfw.wmnet with OS bookworm [production]
21:59 <maryum> Deployed security fix for T402313 [production]
21:47 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P82069 and previous config saved to /var/cache/conftool/dbconfig/20250828-214737-ladsgroup.json [production]
21:32 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T402925)', diff saved to https://phabricator.wikimedia.org/P82068 and previous config saved to /var/cache/conftool/dbconfig/20250828-213230-ladsgroup.json [production]
21:26 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db1212 (T402925)', diff saved to https://phabricator.wikimedia.org/P82067 and previous config saved to /var/cache/conftool/dbconfig/20250828-212609-ladsgroup.json [production]
21:26 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
21:25 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1212.eqiad.wmnet with reason: Maintenance [production]
21:25 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T402925)', diff saved to https://phabricator.wikimedia.org/P82066 and previous config saved to /var/cache/conftool/dbconfig/20250828-212528-ladsgroup.json [production]
21:10 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P82065 and previous config saved to /var/cache/conftool/dbconfig/20250828-211021-ladsgroup.json [production]
20:55 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P82064 and previous config saved to /var/cache/conftool/dbconfig/20250828-205513-ladsgroup.json [production]
20:50 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=99) [admin]
20:40 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T402925)', diff saved to https://phabricator.wikimedia.org/P82063 and previous config saved to /var/cache/conftool/dbconfig/20250828-204006-ladsgroup.json [production]
20:34 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Depooling db1198 (T402925)', diff saved to https://phabricator.wikimedia.org/P82062 and previous config saved to /var/cache/conftool/dbconfig/20250828-203413-ladsgroup.json [production]
20:34 <ladsgroup@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1198.eqiad.wmnet with reason: Maintenance [production]
20:33 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1189 (T402925)', diff saved to https://phabricator.wikimedia.org/P82061 and previous config saved to /var/cache/conftool/dbconfig/20250828-203350-ladsgroup.json [production]
20:20 <dancy@deploy1003> Finished scap sync-world: Backport for [[gerrit:1182857|Set $wgPHPSessionHandling to 'disable' on group0 wikis (T362324)]] (duration: 13m 56s) [production]
20:18 <ladsgroup@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P82060 and previous config saved to /var/cache/conftool/dbconfig/20250828-201843-ladsgroup.json [production]
20:15 <dancy@deploy1003> hokwelum, dancy: Continuing with sync [production]
20:10 <dancy@deploy1003> hokwelum, dancy: Backport for [[gerrit:1182857|Set $wgPHPSessionHandling to 'disable' on group0 wikis (T362324)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
20:06 <dancy@deploy1003> Started scap sync-world: Backport for [[gerrit:1182857|Set $wgPHPSessionHandling to 'disable' on group0 wikis (T362324)]] [production]