801-850 of 10000 results (105ms)
2025-11-14 §
13:48 <jclark@cumin1003> START - Cookbook sre.network.configure-switch-interfaces for host wdqs1029 [production]
13:48 <jclark@cumin1003> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host wdqs1028 [production]
13:48 <jclark@cumin1003> START - Cookbook sre.network.configure-switch-interfaces for host wdqs1028 [production]
13:47 <jclark@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1028.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
13:46 <jclark@cumin1003> START - Cookbook sre.dns.netbox [production]
13:45 <jclark@cumin1003> START - Cookbook sre.hosts.provision for host wdqs1028.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
13:39 <jclark@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
13:39 <jclark@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt wdqs test servers - jclark@cumin1003" [production]
13:39 <jclark@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: added network and mgmt wdqs test servers - jclark@cumin1003" [production]
13:35 <jclark@cumin1003> START - Cookbook sre.dns.netbox [production]
13:02 <cmooney@cumin1003> END (ERROR) - Cookbook sre.network.peering (exit_code=97) with action 'configure' for AS: 45102 [production]
13:02 <cmooney@cumin1003> START - Cookbook sre.network.peering with action 'configure' for AS: 45102 [production]
12:47 <moritzm> rebalance eqiad/D following switch migration T405945 [production]
11:43 <fceratto@cumin1003> END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:40 <fceratto@cumin1003> START - Cookbook sre.mysql.clone of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:39 <fceratto@cumin1003> END (ERROR) - Cookbook sre.mysql.clone (exit_code=97) of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:39 <fceratto@cumin1003> START - Cookbook sre.mysql.clone of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:37 <fceratto@cumin1003> END (ERROR) - Cookbook sre.mysql.clone (exit_code=97) of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:36 <fceratto@cumin1003> START - Cookbook sre.mysql.clone of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:29 <fceratto@cumin1003> END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:17 <fceratto@cumin1003> START - Cookbook sre.mysql.clone of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:16 <fceratto@cumin1003> END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:09 <fceratto@cumin1003> START - Cookbook sre.mysql.clone of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:04 <fceratto@cumin1003> END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:04 <fceratto@cumin1003> START - Cookbook sre.mysql.clone of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:03 <fceratto@cumin1003> END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
11:02 <fceratto@cumin1003> START - Cookbook sre.mysql.clone of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
10:56 <fceratto@cumin1003> END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
10:48 <fceratto@cumin1003> START - Cookbook sre.mysql.clone of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
10:47 <fceratto@cumin1003> END (FAIL) - Cookbook sre.mysql.clone (exit_code=99) of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
10:38 <fceratto@cumin1003> START - Cookbook sre.mysql.clone of db2230.codfw.wmnet onto db-test1001.eqiad.wmnet [production]
10:37 <bwojtowicz@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' . [production]
10:09 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [production]
10:08 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [production]
09:20 <marostegui@cumin1003> conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet,service=s4 [production]
08:04 <arnaudb@cumin1003> END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Security Update [production]
06:58 <arnaudb@cumin1003> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Security Update [production]
06:54 <moritzm> rebalance eqiad/C following switch migration T405945 [production]
06:27 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1024].eqiad.wmnet with reason: Cloning clouddb1024:s4 [production]
06:25 <marostegui@cumin1003> conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet,service=s4 [production]
01:14 <mwpresync@deploy2002> Finished scap build-images: Publishing wmf/next image (duration: 13m 46s) [production]
01:00 <mwpresync@deploy2002> Started scap build-images: Publishing wmf/next image [production]
2025-11-13 §
23:31 <denisse@deploy2002> Finished deploy [librenms/librenms@5fca3ff]: Upgrade LibreNMS to 25.10.0 - T410039 (duration: 00m 15s) [production]
23:30 <denisse@deploy2002> Started deploy [librenms/librenms@5fca3ff]: Upgrade LibreNMS to 25.10.0 - T410039 [production]
23:08 <catrope@deploy2002> Finished scap sync-world: Backport for [[gerrit:1200173|Drop references to removed Advanced mobile contribution configuration (T402470)]] (duration: 09m 30s) [production]
23:04 <catrope@deploy2002> catrope, jdlrobson: Continuing with sync [production]
23:01 <catrope@deploy2002> catrope, jdlrobson: Backport for [[gerrit:1200173|Drop references to removed Advanced mobile contribution configuration (T402470)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
22:59 <catrope@deploy2002> Started scap sync-world: Backport for [[gerrit:1200173|Drop references to removed Advanced mobile contribution configuration (T402470)]] [production]
22:28 <jhathaway@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on sretest2001.codfw.wmnet with reason: sleep test [production]
21:53 <kharlan@deploy2002> Finished scap sync-world: Backport for [[gerrit:1204966|hCaptcha: Simplify ConfirmEditTriggersCaptcha logic for API edits (T405586)]] (duration: 08m 56s) [production]