4751-4800 of 10000 results (66ms)
2025-03-17 §
08:23 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/changeprop: sync [production]
07:29 <marostegui@cumin2002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:28 <marostegui@cumin2002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
07:17 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:17 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
07:16 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 [production]
07:15 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 [production]
07:14 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:13 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
2025-03-16 §
14:08 <claime> sudo postqueue -j | jq -r 'select(.sender == "vrts-bounce@wikimedia.org") | .queue_id' | sudo postsuper -d - # mx-out1001 [production]
13:59 <Emperor> sudo postqueue -j | jq -r ' select(.recipients[0].address == "vrts-bounce@wikimedia.org") | select(.recipients[1].address == null) | .queue_id' | sudo postsuper -d - # mx-in2001 [production]
13:39 <Emperor> restart postfix on mx-in2001 [production]
11:42 <taavi@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-38 [tools]
11:37 <taavi@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-38 [tools]
07:25 <wmbot~melos@tools-bastion-13> SULWatcher/manage.sh restart # SULWatchers disconnected [tools.stewardbots]
2025-03-15 §
18:11 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2046.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
18:00 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host ganeti2046.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:31 <wmbot~dcaro@urcuchillay> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 (T388965) [tools]
15:14 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16, tools-k8s-worker-nfs-34, tools-k8s-worker-nfs-77 (T388965) [tools]
15:14 <wmbot~dcaro@urcuchillay> END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 (T388965) [tools]
15:14 <wmbot~dcaro@urcuchillay> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-16,tools-k8s-worker-nfs-34,tools-k8s-worker-nfs-77 (T388965) [tools]
12:55 <dcaro> there was an NFS hiccup that made the NFS checks fail for a second and some workers get stuck for a bit T388965 [tools]
11:22 <wmbot~lucaswerkmeister@tools-bastion-13> deployed ab6621b22d (l10n updates: ar) [tools.lexeme-forms]
00:51 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2046 [production]
00:51 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host ganeti2046 [production]
2025-03-14 §
21:36 <Reedy> deployed https://gerrit.wikimedia.org/r/1127982 [releng]
21:25 <zabe> zabe@mwmaint2002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php testwiki --delete /home/zabe/afl_text_table_deletedump/testwiki --sleep 0.3 # T381599 [production]
21:06 <zabe> zabe@mwmaint2002:~$ mwscript extensions/AbuseFilter/maintenance/MigrateESRefToAflTable.php testwiki --dump /home/zabe/afl_text_table_dump/testwiki --deletedump /home/zabe/afl_text_table_deletedump/testwiki --sleep 0.3 # T381599 [production]
16:55 <Lucas_WMDE> manually killed job https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81/2928/console which had been stuck since 16:33 UTC, blocking gate-and-submit :( [releng]
16:51 <root@deploy2002> helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [production]
16:51 <root@deploy2002> helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. [production]
16:15 <slyngshede@cumin1002> DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Vivian Rook out of all services on: 2288 hosts [production]
16:14 <slyngshede@cumin1002> DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Vivian Rook out of all services on: 2288 hosts [production]
16:05 <sukhe@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling A:liberica-canary [production]
16:05 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin pooling A:liberica-canary [production]
16:04 <sukhe@cumin1002> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling A:liberica-canary [production]
16:04 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin depooling A:liberica-canary [production]
16:04 <sukhe@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling A:liberica-canary [production]
16:03 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin pooling A:liberica-canary [production]
16:01 <sukhe@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling A:liberica-canary [production]
16:00 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin pooling A:liberica-canary [production]
16:00 <sukhe@cumin1002> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling A:liberica-canary [production]
16:00 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin depooling A:liberica-canary [production]
15:50 <slyngshede@cumin1002> DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Vivian Rook out of all services on: 2288 hosts [production]
15:41 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2075.codfw.wmnet with OS bullseye [production]
15:41 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be2075.codfw.wmnet with OS bullseye [production]
15:37 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2075.codfw.wmnet with OS bullseye [production]
15:20 <root@deploy2002> helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [production]
15:20 <root@deploy2002> helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. [production]
15:20 <root@deploy2002> helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [production]