1-50 of 10000 results (88ms)
2026-02-11 §
03:51 <ryankemper@cumin2002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-jumbo-eqiad [production]
02:04 <ryankemper> `[kafka-jumbo]`T411568 Rebooting kafka-jumbo one host at a time (sec updates) [production]
02:03 <ryankemper@cumin2002> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-jumbo-eqiad [production]
02:01 <mwpresync@deploy2002> Finished scap build-images: Publishing wmf/next image (duration: 00m 25s) [production]
02:00 <mwpresync@deploy2002> Started scap build-images: Publishing wmf/next image [production]
00:33 <brett> Import libvmod-re2 2.0.0-2~deb13+wmf2 into trixie-wikimedia - T401832 [production]
00:08 <ryankemper@cumin2002> END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling reboot on A:kafka-test-eqiad [production]
2026-02-10 §
23:32 <ryankemper@cumin2002> START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling reboot on A:kafka-test-eqiad [production]
23:04 <ryankemper> `[opensearch-ipoid]` T416345 Restarted (~16 mins ago) `opensearch-ipoid-masters-1` to force it to schedule to a k8s node with 10G networking, hopefully this helps w/ latency [production]
23:03 <dreamyjazz@deploy2002> Finished scap sync-world: Backport for [[gerrit:1237898|Stop writing old for CheckUser user agent table migration on group1 (T361206)]] (duration: 09m 05s) [production]
23:02 <jasmine_> "homer on lsw*codfw* to remove bgp for wikikube-workers T409104" [production]
22:59 <dreamyjazz@deploy2002> dreamyjazz: Continuing with sync [production]
22:57 <dreamyjazz@deploy2002> dreamyjazz: Backport for [[gerrit:1237898|Stop writing old for CheckUser user agent table migration on group1 (T361206)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
22:54 <dreamyjazz@deploy2002> Started scap sync-world: Backport for [[gerrit:1237898|Stop writing old for CheckUser user agent table migration on group1 (T361206)]] [production]
22:40 <kharlan@deploy2002> Finished scap sync-world: Backport for [[gerrit:1238436|ConfirmEdit: Add error count threshold for apiUrl health checks (T416817)]] (duration: 12m 34s) [production]
22:34 <kharlan@deploy2002> kharlan: Continuing with sync [production]
22:32 <kharlan@deploy2002> kharlan: Backport for [[gerrit:1238436|ConfirmEdit: Add error count threshold for apiUrl health checks (T416817)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
22:31 <dzahn@cumin2002> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: security release 20260210 [production]
22:28 <kharlan@deploy2002> Started scap sync-world: Backport for [[gerrit:1238436|ConfirmEdit: Add error count threshold for apiUrl health checks (T416817)]] [production]
22:23 <dzahn@cumin2002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: security release 20260210 [production]
22:21 <dzahn@cumin2002> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: security release 20260210 [production]
22:12 <dzahn@cumin2002> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: security release 20260210 [production]
21:58 <kemayo@deploy2002> Finished scap sync-world: Backport for [[gerrit:1238430|Edit check suggestions beta-feature in allowlist (T399611)]], [[gerrit:1238049|Turn on Parsoid read views by default on labs (take 2) (T357054)]], [[gerrit:1238427|Update title / desc of Special:LintTemplateErrors (T170874)]] (duration: 39m 46s) [production]
21:46 <kemayo@deploy2002> cscott, arlolra, kemayo: Continuing with sync [production]
21:44 <kemayo@deploy2002> cscott, arlolra, kemayo: Backport for [[gerrit:1238430|Edit check suggestions beta-feature in allowlist (T399611)]], [[gerrit:1238049|Turn on Parsoid read views by default on labs (take 2) (T357054)]], [[gerrit:1238427|Update title / desc of Special:LintTemplateErrors (T170874)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
21:36 <xcollazo@deploy2002> Finished deploy [analytics/refinery@f84b3cc] (thin): Regular analytics weekly train THIN [analytics/refinery@f84b3cc2] (duration: 01m 55s) [production]
21:34 <xcollazo@deploy2002> Started deploy [analytics/refinery@f84b3cc] (thin): Regular analytics weekly train THIN [analytics/refinery@f84b3cc2] [production]
21:32 <xcollazo@deploy2002> Finished deploy [analytics/refinery@f84b3cc]: Regular analytics weekly train [analytics/refinery@f84b3cc2] (duration: 04m 59s) [production]
21:27 <xcollazo@deploy2002> Started deploy [analytics/refinery@f84b3cc]: Regular analytics weekly train [analytics/refinery@f84b3cc2] [production]
21:23 <xcollazo@deploy2002> Finished deploy [analytics/refinery@f84b3cc] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f84b3cc2] (duration: 01m 57s) [production]
21:21 <xcollazo@deploy2002> Started deploy [analytics/refinery@f84b3cc] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f84b3cc2] [production]
21:19 <kemayo@deploy2002> Started scap sync-world: Backport for [[gerrit:1238430|Edit check suggestions beta-feature in allowlist (T399611)]], [[gerrit:1238049|Turn on Parsoid read views by default on labs (take 2) (T357054)]], [[gerrit:1238427|Update title / desc of Special:LintTemplateErrors (T170874)]] [production]
21:13 <kemayo@deploy2002> Finished scap sync-world: Backport for [[gerrit:1238382|Edit check: turn off the addReference a/b test on enwiki (T367343 T416378)]], [[gerrit:1238383|Edit check: enable suggestions beta-feature on enwiki (T399611)]] (duration: 09m 58s) [production]
21:09 <kemayo@deploy2002> kemayo: Continuing with sync [production]
21:07 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-research: apply [production]
21:06 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-research: apply [production]
21:05 <kemayo@deploy2002> kemayo: Backport for [[gerrit:1238382|Edit check: turn off the addReference a/b test on enwiki (T367343 T416378)]], [[gerrit:1238383|Edit check: enable suggestions beta-feature on enwiki (T399611)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
21:03 <kemayo@deploy2002> Started scap sync-world: Backport for [[gerrit:1238382|Edit check: turn off the addReference a/b test on enwiki (T367343 T416378)]], [[gerrit:1238383|Edit check: enable suggestions beta-feature on enwiki (T399611)]] [production]
19:58 <jasmine@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wikikube-worker2241.codfw.wmnet [production]
19:58 <jasmine@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:58 <jasmine@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker2241.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jasmine@cumin2002" [production]
19:58 <jasmine@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker2241.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jasmine@cumin2002" [production]
19:56 <ryankemper@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on an-worker[1132,1148].eqiad.wmnet with reason: T411919 [production]
19:54 <jasmine@cumin2002> START - Cookbook sre.dns.netbox [production]
19:48 <sukhe@cumin1003> END (PASS) - Cookbook sre.dns.admin (exit_code=0) DNS admin: pool site magru [reason: no reason specified, ] [production]
19:48 <sukhe@cumin1003> START - Cookbook sre.dns.admin DNS admin: pool site magru [reason: no reason specified, ] [production]
19:48 <jasmine@cumin2002> START - Cookbook sre.hosts.decommission for hosts wikikube-worker2241.codfw.wmnet [production]
19:48 <jasmine@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wikikube-worker[2236-2240].codfw.wmnet [production]
19:48 <jasmine@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:48 <jasmine@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wikikube-worker[2236-2240].codfw.wmnet decommissioned, removing all IPs except the asset tag one - jasmine@cumin2002" [production]