4701-4750 of 10000 results (137ms)
2025-09-26 §
07:00 <jelto@cumin1003> START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab replica [production]
02:59 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1230.eqiad.wmnet with OS bullseye [production]
02:59 <jclark@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
02:59 <jclark@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
02:58 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1232.eqiad.wmnet with OS bullseye [production]
02:58 <jclark@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
02:55 <jclark@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
02:38 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host an-worker1231.eqiad.wmnet with OS bullseye [production]
02:38 <jclark@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
02:38 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1232.eqiad.wmnet with reason: host reimage [production]
02:38 <jclark@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002" [production]
02:37 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1229.eqiad.wmnet with OS bullseye [production]
02:36 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1226.eqiad.wmnet with OS bullseye [production]
02:36 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1225.eqiad.wmnet with OS bullseye [production]
02:36 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1227.eqiad.wmnet with OS bullseye [production]
02:34 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1230.eqiad.wmnet with reason: host reimage [production]
02:31 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1232.eqiad.wmnet with reason: host reimage [production]
02:31 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1230.eqiad.wmnet with reason: host reimage [production]
02:20 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1225.eqiad.wmnet with OS bullseye [production]
02:19 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1226.eqiad.wmnet with OS bullseye [production]
02:19 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1227.eqiad.wmnet with OS bullseye [production]
02:19 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1228.eqiad.wmnet with OS bullseye [production]
02:16 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1229.eqiad.wmnet with OS bullseye [production]
02:16 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1232.eqiad.wmnet with OS bullseye [production]
02:16 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1230.eqiad.wmnet with OS bullseye [production]
02:14 <jclark@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host an-worker1232.eqiad.wmnet with OS bullseye [production]
02:14 <jclark@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on an-worker1231.eqiad.wmnet with reason: host reimage [production]
02:13 <krinkle@deploy2002> Finished scap sync-world: Backport for [[gerrit:1191553|Disable inert MobileFrontend on wikimedia.org wikis lacking DNS (part 2) (T152882)]] (duration: 13m 09s) [production]
02:10 <jclark@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on an-worker1231.eqiad.wmnet with reason: host reimage [production]
02:08 <krinkle@deploy2002> krinkle: Continuing with sync [production]
02:07 <krinkle@deploy2002> krinkle: Backport for [[gerrit:1191553|Disable inert MobileFrontend on wikimedia.org wikis lacking DNS (part 2) (T152882)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
02:00 <krinkle@deploy2002> Started scap sync-world: Backport for [[gerrit:1191553|Disable inert MobileFrontend on wikimedia.org wikis lacking DNS (part 2) (T152882)]] [production]
01:51 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1231.eqiad.wmnet with OS bullseye [production]
01:38 <jclark@cumin1002> START - Cookbook sre.hosts.reimage for host an-worker1232.eqiad.wmnet with OS bullseye [production]
01:15 <mwpresync@deploy2002> Finished scap build-images: Publishing wmf/next image (duration: 13m 53s) [production]
01:01 <mwpresync@deploy2002> Started scap build-images: Publishing wmf/next image [production]
00:36 <rzl@deploy1003> helmfile [codfw] DONE helmfile.d/services/rest-gateway: apply [production]
00:36 <rzl@deploy1003> helmfile [codfw] START helmfile.d/services/rest-gateway: apply [production]
00:33 <rzl@deploy1003> helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply [production]
00:32 <rzl@deploy1003> helmfile [eqiad] START helmfile.d/services/rest-gateway: apply [production]
00:10 <rzl@deploy1003> helmfile [codfw] DONE helmfile.d/services/api-gateway: apply [production]
00:10 <rzl@deploy1003> helmfile [codfw] START helmfile.d/services/api-gateway: apply [production]
00:03 <rzl@deploy1003> helmfile [eqiad] DONE helmfile.d/services/api-gateway: apply [production]
00:03 <rzl@deploy1003> helmfile [eqiad] START helmfile.d/services/api-gateway: apply [production]
2025-09-25 §
23:29 <mutante> releases2003 - re-enabling puppet which was disabled for debugging T405352 - then the deployment server failover happened and this server didn't get the update what the active deployment server was.. which subsequently caused T405646 [production]
23:10 <rzl@deploy1003> helmfile [staging] DONE helmfile.d/services/rest-gateway: apply [production]
23:10 <rzl@deploy1003> helmfile [staging] START helmfile.d/services/rest-gateway: apply [production]
23:08 <rzl@deploy1003> helmfile [staging] DONE helmfile.d/services/api-gateway: apply [production]
23:08 <rzl@deploy1003> helmfile [staging] START helmfile.d/services/api-gateway: apply [production]
22:44 <eileen> civicrm upgraded from a53165c1 to 3771abe2 [production]