351-400 of 10000 results (39ms)
2026-11-06 §
11:48 <btullis@cumin1004> START - Cookbook sre.hosts.reboot-single for host wdqs1032.eqiad.wmnet [production]
11:47 <btullis@cumin1004> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1032.eqiad.wmnet [production]
11:43 <btullis@cumin1004> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs2025.codfw.wmnet [production]
11:43 <btullis@cumin1004> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs2025.codfw.wmnet [production]
11:42 <btullis@cumin1004> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1031.eqiad.wmnet [production]
11:42 <btullis@cumin1004> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1031.eqiad.wmnet [production]
11:39 <fabfur> installing haproxykafka 0.4.0 on cp7001 for testing (T443632) (T443640) [production]
11:36 <a-pizzata@deploy2003> Finished deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] (duration: 04m 50s) [production]
11:32 <btullis@cumin1004> START - Cookbook sre.hosts.reboot-single for host wdqs2025.codfw.wmnet [production]
11:32 <btullis@cumin1004> START - Cookbook sre.hosts.reboot-single for host wdqs1031.eqiad.wmnet [production]
11:31 <btullis@cumin1004> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet [production]
11:31 <a-pizzata@deploy2003> Started deploy [analytics/refinery@f4aa212]: Regular analytics weekly train [analytics/refinery@f4aa2127] [production]
11:30 <a-pizzata@deploy2003> Finished deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] (duration: 00m 46s) [production]
11:30 <a-pizzata@deploy2003> Started deploy [analytics/refinery@f4aa212] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@f4aa2127] [production]
11:19 <btullis@cumin1004> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1031.eqiad.wmnet [production]
11:18 <btullis@cumin1004> END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts wdqs1030.eqiad.wmnet [production]
11:18 <btullis@cumin1004> END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host wdqs1030.eqiad.wmnet [production]
11:09 <btullis@cumin1004> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts wdqs2025.codfw.wmnet [production]
11:09 <btullis@cumin1004> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs2025.codfw.wmnet [production]
11:08 <btullis@cumin1004> START - Cookbook sre.hosts.reboot-single for host wdqs1030.eqiad.wmnet [production]
11:00 <btullis@cumin1004> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts wdqs1030.eqiad.wmnet [production]
10:49 <jiji@cumin1004> END (PASS) - Cookbook sre.discovery.service-route (exit_code=0) depool kartotherian in eqiad: maintenance [production]
10:44 <jiji@cumin1004> START - Cookbook sre.discovery.service-route depool kartotherian in eqiad: maintenance [production]
10:42 <effie> depool kartotherian from eqiad (ie maps) - T443438 [production]
10:34 <jelto@deploy2003> Finished scap deploy-service miscweb-aux: Deploy to fix T443357 (duration: 03m 39s) [production]
10:33 <jelto@deploy2003> scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb [production]
10:31 <jelto@deploy2003> scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb [production]
10:31 <jelto@deploy2003> Started scap deploy-service miscweb-aux: Deploy to fix T443357 [production]
10:29 <marostegui> Limit icinga db user connections to 10 in ms* and pc* T442995 [production]
10:16 <ayounsi@cumin1004> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox [production]
10:16 <ayounsi@cumin1004> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox [production]
10:16 <ayounsi@cumin1004> END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary [production]
10:11 <fabfur@cumin1004> conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox [production]
10:08 <ayounsi@cumin1004> START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary [production]
09:04 <mvernon@cumin1004> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe [production]
08:57 <jelto@deploy2003> Finished scap deploy-service miscweb-aux: Deploy to fix T443826 (duration: 04m 11s) [production]
08:56 <jelto@deploy2003> scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb [production]
08:53 <jelto@deploy2003> scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb [production]
08:53 <jelto@deploy2003> Started scap deploy-service miscweb-aux: Deploy to fix T443826 [production]
08:52 <mvernon@cumin2003> END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe [production]
08:48 <mvernon@cumin1004> START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe [production]
08:47 <mvernon@cumin2003> START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe [production]
06:52 <XioNoX> push pfw policies - T443852 [production]
00:00 <root@deploy2003> helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync [production]
00:00 <root@deploy2003> helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync [production]
2026-11-05 §
23:30 <mutante> codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout. [production]
23:18 <maryum> Deployed security fix for T443424, T443467, T443036 [production]
23:10 <ladsgroup@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance [production]
23:10 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json [production]
23:00 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json [production]