|
2026-02-10
ยง
|
| 15:10 |
<oblivian@cumin1003> |
END (PASS) - Cookbook sre.deploy.hiddenparma (exit_code=0) Hiddenparma deployment to the alerting hosts with reason: "Fix mysql has gone away error in hp - oblivian@cumin1003" |
[production] |
| 15:10 |
<oblivian@cumin1003> |
END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix mysql has gone away error in hp - oblivian@cumin1003 |
[production] |
| 15:10 |
<oblivian@cumin1003> |
START - Cookbook sre.deploy.python-code hiddenparma to alert[1002,2002].wikimedia.org with reason: Fix mysql has gone away error in hp - oblivian@cumin1003 |
[production] |
| 15:09 |
<oblivian@cumin1003> |
START - Cookbook sre.deploy.hiddenparma Hiddenparma deployment to the alerting hosts with reason: "Fix mysql has gone away error in hp - oblivian@cumin1003" |
[production] |
| 15:08 |
<fceratto@cumin1003> |
END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) |
[production] |
| 15:08 |
<fceratto@cumin1003> |
START - Cookbook sre.mysql.update-replication |
[production] |
| 15:08 |
<fceratto@cumin1003> |
END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) |
[production] |
| 15:08 |
<fceratto@cumin1003> |
START - Cookbook sre.mysql.update-replication |
[production] |
| 15:08 |
<dpogorzelski@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-staging-etcd2003.codfw.wmnet with reason: host reimage |
[production] |
| 15:07 |
<fceratto@cumin1003> |
END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) |
[production] |
| 15:07 |
<fceratto@cumin1003> |
START - Cookbook sre.mysql.update-replication |
[production] |
| 15:04 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudgw1004.eqiad.wmnet with OS trixie |
[production] |
| 15:02 |
<XioNoX> |
cr2-drmrs> request vmhost reboot - T416441 |
[production] |
| 14:55 |
<fceratto@cumin1003> |
END (ERROR) - Cookbook sre.mysql.update-replication (exit_code=97) |
[production] |
| 14:55 |
<fceratto@cumin1003> |
START - Cookbook sre.mysql.update-replication |
[production] |
| 14:53 |
<dpogorzelski@cumin1003> |
START - Cookbook sre.hosts.reimage for host ml-staging-etcd2003.codfw.wmnet with OS bookworm |
[production] |
| 14:53 |
<fceratto@cumin1003> |
END (FAIL) - Cookbook sre.mysql.update-replication (exit_code=99) |
[production] |
| 14:53 |
<fceratto@cumin1003> |
START - Cookbook sre.mysql.update-replication |
[production] |
| 14:51 |
<ayounsi@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on cr2-drmrs,cr2-drmrs IPv6,cr2-drmrs.mgmt with reason: router upgrade |
[production] |
| 14:47 |
<dpogorzelski@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ml-staging-etcd2001.codfw.wmnet with OS bookworm |
[production] |
| 14:43 |
<andrew@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudgw1004.eqiad.wmnet with reason: host reimage |
[production] |
| 14:41 |
<XioNoX> |
asw1-b12-drmrs> request system reboot - T416441 |
[production] |
| 14:40 |
<ayounsi@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 20 hosts with reason: Switch upgrade |
[production] |
| 14:37 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on cloudgw1004.eqiad.wmnet with reason: host reimage |
[production] |
| 14:24 |
<XioNoX> |
cr1-drmrs> request vmhost reboot - T416441 |
[production] |
| 14:20 |
<moritzm> |
installing wmf-certificates 0_~20260209 on bookworm hosts T415255 |
[production] |
| 14:20 |
<andrew@cumin2002> |
START - Cookbook sre.hosts.reimage for host cloudgw1004.eqiad.wmnet with OS trixie |
[production] |
| 14:19 |
<sukhe@puppetserver1001> |
conftool action : set/pooled=no; selector: cluster=dnsbox,dc=drmrs [reason: drmrs maintenance] |
[production] |
| 14:05 |
<ayounsi@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cr1-drmrs,cr1-drmrs IPv6,cr1-drmrs.mgmt with reason: router upgrade |
[production] |
| 14:03 |
<fceratto@cumin1003> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dborch1002.wikimedia.org with OS trixie |
[production] |
| 14:02 |
<ayounsi@cumin1003> |
END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: depool site drmrs [reason: no reason specified, ] |
[production] |
| 14:02 |
<ayounsi@cumin1003> |
START - Cookbook sre.dns.admin DNS admin: depool site drmrs [reason: no reason specified, ] |
[production] |
| 13:59 |
<jforrester@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1238338|EmailAuthHookHandler: Check if WikimediaEvents loaded before using WikimediaEventsCountryCodeLookup (T416983)]], [[gerrit:1238339|More robust SkinTemplateNavigation hook handler (T416978)]] (duration: 07m 55s) |
[production] |
| 13:54 |
<jforrester@deploy2002> |
jforrester: Continuing with sync |
[production] |
| 13:53 |
<jforrester@deploy2002> |
jforrester: Backport for [[gerrit:1238338|EmailAuthHookHandler: Check if WikimediaEvents loaded before using WikimediaEventsCountryCodeLookup (T416983)]], [[gerrit:1238339|More robust SkinTemplateNavigation hook handler (T416978)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 13:51 |
<jforrester@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1238338|EmailAuthHookHandler: Check if WikimediaEvents loaded before using WikimediaEventsCountryCodeLookup (T416983)]], [[gerrit:1238339|More robust SkinTemplateNavigation hook handler (T416978)]] |
[production] |
| 13:50 |
<moritzm> |
upload wmf-certificates 0_~20260209 to bookworm-wikimedia/main (drops the Puppet 5 CA from the cert bundle) T415255 |
[production] |
| 13:48 |
<fceratto@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dborch1002.wikimedia.org with reason: host reimage |
[production] |
| 13:42 |
<fceratto@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on dborch1002.wikimedia.org with reason: host reimage |
[production] |
| 13:29 |
<fceratto@cumin1003> |
START - Cookbook sre.hosts.reimage for host dborch1002.wikimedia.org with OS trixie |
[production] |
| 13:20 |
<ladsgroup@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1238332|Page: Bump the size of pdf thumbnail size to a standard size (T416620)]] (duration: 08m 12s) |
[production] |
| 13:18 |
<dpogorzelski@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ml-staging-etcd2001.codfw.wmnet with reason: host reimage |
[production] |
| 13:16 |
<ladsgroup@deploy2002> |
ladsgroup: Continuing with sync |
[production] |
| 13:14 |
<ladsgroup@deploy2002> |
ladsgroup: Backport for [[gerrit:1238332|Page: Bump the size of pdf thumbnail size to a standard size (T416620)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 13:13 |
<dpogorzelski@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ml-staging-etcd2001.codfw.wmnet with reason: host reimage |
[production] |
| 13:12 |
<ladsgroup@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1238332|Page: Bump the size of pdf thumbnail size to a standard size (T416620)]] |
[production] |
| 12:57 |
<dpogorzelski@cumin1003> |
START - Cookbook sre.hosts.reimage for host ml-staging-etcd2001.codfw.wmnet with OS bookworm |
[production] |
| 12:50 |
<kharlan@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1238327|Revert "IPReputation: Switch to OpenSearch backend" (T416164)]] (duration: 06m 40s) |
[production] |
| 12:46 |
<kharlan@deploy2002> |
kharlan: Continuing with sync |
[production] |
| 12:46 |
<kharlan@deploy2002> |
kharlan: Backport for [[gerrit:1238327|Revert "IPReputation: Switch to OpenSearch backend" (T416164)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |