3701-3750 of 10000 results (155ms)
2025-03-17 §
08:30 <moritzm> updated bookworm installer image to Bookworm 12.10 T389034 [production]
08:23 <elukey@deploy2002> helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: sync [production]
08:23 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/changeprop-jobqueue: sync [production]
08:23 <elukey@deploy2002> helmfile [staging] DONE helmfile.d/services/changeprop: sync [production]
08:23 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/changeprop: sync [production]
07:29 <marostegui@cumin2002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:28 <marostegui@cumin2002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
07:17 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:17 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
07:16 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section test-s4 [production]
07:15 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section test-s4 [production]
07:14 <marostegui@cumin1002> END (PASS) - Cookbook sre.switchdc.databases.prepare (exit_code=0) for the switch from codfw to eqiad for section x1 [production]
07:13 <marostegui@cumin1002> START - Cookbook sre.switchdc.databases.prepare for the switch from codfw to eqiad for section x1 [production]
2025-03-16 §
14:08 <claime> sudo postqueue -j | jq -r 'select(.sender == "vrts-bounce@wikimedia.org") | .queue_id' | sudo postsuper -d - # mx-out1001 [production]
13:59 <Emperor> sudo postqueue -j | jq -r ' select(.recipients[0].address == "vrts-bounce@wikimedia.org") | select(.recipients[1].address == null) | .queue_id' | sudo postsuper -d - # mx-in2001 [production]
13:39 <Emperor> restart postfix on mx-in2001 [production]
2025-03-15 §
18:11 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti2046.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
18:00 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host ganeti2046.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
00:51 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ganeti2046 [production]
00:51 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host ganeti2046 [production]
2025-03-14 §
21:25 <zabe> zabe@mwmaint2002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php testwiki --delete /home/zabe/afl_text_table_deletedump/testwiki --sleep 0.3 # T381599 [production]
21:06 <zabe> zabe@mwmaint2002:~$ mwscript extensions/AbuseFilter/maintenance/MigrateESRefToAflTable.php testwiki --dump /home/zabe/afl_text_table_dump/testwiki --deletedump /home/zabe/afl_text_table_deletedump/testwiki --sleep 0.3 # T381599 [production]
16:51 <root@deploy2002> helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [production]
16:51 <root@deploy2002> helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. [production]
16:15 <slyngshede@cumin1002> DONE (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Vivian Rook out of all services on: 2288 hosts [production]
16:14 <slyngshede@cumin1002> DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Vivian Rook out of all services on: 2288 hosts [production]
16:05 <sukhe@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling A:liberica-canary [production]
16:05 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin pooling A:liberica-canary [production]
16:04 <sukhe@cumin1002> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling A:liberica-canary [production]
16:04 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin depooling A:liberica-canary [production]
16:04 <sukhe@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling A:liberica-canary [production]
16:03 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin pooling A:liberica-canary [production]
16:01 <sukhe@cumin1002> END (FAIL) - Cookbook sre.loadbalancer.admin (exit_code=1) pooling A:liberica-canary [production]
16:00 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin pooling A:liberica-canary [production]
16:00 <sukhe@cumin1002> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) depooling A:liberica-canary [production]
16:00 <sukhe@cumin1002> START - Cookbook sre.loadbalancer.admin depooling A:liberica-canary [production]
15:50 <slyngshede@cumin1002> DONE (FAIL) - Cookbook sre.idm.logout (exit_code=99) Logging Vivian Rook out of all services on: 2288 hosts [production]
15:41 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2075.codfw.wmnet with OS bullseye [production]
15:41 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host ms-be2075.codfw.wmnet with OS bullseye [production]
15:37 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host ms-be2075.codfw.wmnet with OS bullseye [production]
15:20 <root@deploy2002> helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [production]
15:20 <root@deploy2002> helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. [production]
15:20 <root@deploy2002> helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [production]
15:20 <root@deploy2002> helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. [production]
15:19 <root@deploy2002> helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [production]
15:19 <root@deploy2002> helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. [production]
15:19 <root@deploy2002> helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'sync'. [production]
15:18 <root@deploy2002> helmfile [aux-k8s-codfw] START helmfile.d/admin 'sync'. [production]
14:55 <herron> kafka-logging reduce mediawiki.httpd.accesslog topic retention from 172800000ms (2d) to 129600000ms (1.5d) [production]
13:33 <kevinbazira@deploy2002> helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-models' for release 'main' . [production]