101-150 of 10000 results (14ms)
2026-11-16 ยง
15:22 <xcollazo@deploy2003> Finished deploy [analytics/refinery@2658f08]: Regular analytics weekly train [analytics/refinery@2658f088] (duration: 05m 48s) [production]
15:17 <jgiannelos@deploy2003> helmfile [staging] DONE helmfile.d/services/thumbor: apply [production]
15:17 <xcollazo@deploy2003> Started deploy [analytics/refinery@2658f08]: Regular analytics weekly train [analytics/refinery@2658f088] [production]
15:16 <jgiannelos@deploy2003> helmfile [staging] START helmfile.d/services/thumbor: apply [production]
15:16 <xcollazo@deploy2003> Finished deploy [analytics/refinery@2658f08] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2658f088] (duration: 00m 50s) [production]
15:16 <xcollazo@deploy2003> Started deploy [analytics/refinery@2658f08] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2658f088] [production]
15:08 <Lucas_WMDE> UTC afternoon backport+config window done [production]
15:04 <ejegg@deploy2003> Finished scap sync-world: Backport for [[gerrit:1369512|Update CentralNotice for preview bugfixes]] (1.48.0-wmf.4) (duration: 19m 07s) [production]
14:59 <sukhe@cumin1004> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [production]
14:56 <sukhe@cumin1004> START - Cookbook sre.loadbalancer.migrate-service-ipip for role: ml_k8s::master@eqiad [production]
14:56 <sukhe@cumin1004> END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for role: ml_k8s::master@codfw [production]
14:56 <sukhe@cumin1004> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [production]
14:54 <sukhe@cumin1004> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [production]
14:52 <ejegg@deploy2003> ejegg: Continuing with deployment [production]
14:50 <sukhe@cumin1004> START - Cookbook sre.loadbalancer.migrate-service-ipip for role: ml_k8s::master@codfw [production]
14:49 <ejegg@deploy2003> ejegg: Backport for [[gerrit:1369512|Update CentralNotice for preview bugfixes]] (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
14:44 <ejegg@deploy2003> Started scap sync-world: Backport for [[gerrit:1369512|Update CentralNotice for preview bugfixes]] (1.48.0-wmf.4) [production]
14:34 <sukhe@cumin1004> START - Cookbook sre.dns.roll-restart-reboot-durum rolling reboot on A:durum [production]
14:32 <lucaswerkmeister-wmde@deploy2003> Finished scap sync-world: Backport for [[gerrit:1369433|dewiki: Add als, bar and nds to import sources (T444759)]] (duration: 16m 14s) [production]
14:26 <moritzm> installing apache2 security updates [production]
14:25 <lucaswerkmeister-wmde@deploy2003> lucaswerkmeister-wmde, tryvix1509: Continuing with deployment [production]
14:20 <lucaswerkmeister-wmde@deploy2003> lucaswerkmeister-wmde, tryvix1509: Backport for [[gerrit:1369433|dewiki: Add als, bar and nds to import sources (T444759)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
14:16 <lucaswerkmeister-wmde@deploy2003> Started scap sync-world: Backport for [[gerrit:1369433|dewiki: Add als, bar and nds to import sources (T444759)]] [production]
14:03 <krinkle@deploy2003> Finished scap sync-world: Backport for [[gerrit:1369599|FileRepo: Switch getDescriptionShortUrl from curid to Special:Redirect (T428296)]] (1.48.0-wmf.4) (duration: 16m 35s) [production]
13:53 <krinkle@deploy2003> krinkle: Continuing with deployment [production]
13:52 <krinkle@deploy2003> krinkle: Backport for [[gerrit:1369599|FileRepo: Switch getDescriptionShortUrl from curid to Special:Redirect (T428296)]] (1.48.0-wmf.4) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
13:50 <Amir1> alter table if exists watchlist engine=innodb,force; on loginwiki (T442779) [production]
13:46 <krinkle@deploy2003> Started scap sync-world: Backport for [[gerrit:1369599|FileRepo: Switch getDescriptionShortUrl from curid to Special:Redirect (T428296)]] (1.48.0-wmf.4) [production]
13:39 <taavi@cumin1004> END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host cloudcumin2001.codfw.wmnet [production]
13:39 <taavi@cumin1004> START - Cookbook sre.hosts.reboot-single for host cloudcumin2001.codfw.wmnet [production]
13:36 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P97467 and previous config saved to /var/cache/conftool/dbconfig/20261116-133630-ladsgroup.json [production]
13:26 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P97466 and previous config saved to /var/cache/conftool/dbconfig/20261116-132628-ladsgroup.json [production]
13:16 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1247 (T441633)', diff saved to https://phabricator.wikimedia.org/P97465 and previous config saved to /var/cache/conftool/dbconfig/20261116-131626-ladsgroup.json [production]
13:08 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Depooling db1247 (T441633)', diff saved to https://phabricator.wikimedia.org/P97464 and previous config saved to /var/cache/conftool/dbconfig/20261116-130852-ladsgroup.json [production]
13:08 <ladsgroup@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1247.eqiad.wmnet with reason: Maintenance [production]
13:08 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1244 (T441633)', diff saved to https://phabricator.wikimedia.org/P97463 and previous config saved to /var/cache/conftool/dbconfig/20261116-130832-ladsgroup.json [production]
12:58 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P97462 and previous config saved to /var/cache/conftool/dbconfig/20261116-125830-ladsgroup.json [production]
12:48 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P97461 and previous config saved to /var/cache/conftool/dbconfig/20261116-124828-ladsgroup.json [production]
12:38 <zabe@deploy2003> Finished scap sync-world: Backport for [[gerrit:1369685|Move reads/writes of wbc_entity_usage to x1 for some s2 wikis (T439753)]] (duration: 37m 12s) [production]
12:38 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1244 (T441633)', diff saved to https://phabricator.wikimedia.org/P97460 and previous config saved to /var/cache/conftool/dbconfig/20261116-123826-ladsgroup.json [production]
12:31 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Depooling db1244 (T441633)', diff saved to https://phabricator.wikimedia.org/P97459 and previous config saved to /var/cache/conftool/dbconfig/20261116-123101-ladsgroup.json [production]
12:31 <ladsgroup@cumin1004> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1244.eqiad.wmnet with reason: Maintenance [production]
12:30 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1242 (T441633)', diff saved to https://phabricator.wikimedia.org/P97458 and previous config saved to /var/cache/conftool/dbconfig/20261116-123041-ladsgroup.json [production]
12:23 <kart_> Updated cxserver to 2026-11-03-143706-production (T428590, T442944) [production]
12:22 <kartik@deploy2003> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
12:22 <zabe@deploy2003> zabe: Continuing with deployment [production]
12:22 <brett@cumin2003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on cp3074.esams.wmnet with reason: troubleshooting cmos failure [production]
12:22 <kartik@deploy2003> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
12:22 <zabe@deploy2003> zabe: Backport for [[gerrit:1369685|Move reads/writes of wbc_entity_usage to x1 for some s2 wikis (T439753)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
12:20 <ladsgroup@cumin1004> dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P97457 and previous config saved to /var/cache/conftool/dbconfig/20261116-122040-ladsgroup.json [production]