|
2026-11-05
ยง
|
| 21:51 |
<mutante> |
ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4 (T440668) |
[production] |
| 21:44 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json |
[production] |
| 21:34 |
<ssastry@deploy2003> |
helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 21:34 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97405 and previous config saved to /var/cache/conftool/dbconfig/20261105-213428-ladsgroup.json |
[production] |
| 21:33 |
<ssastry@deploy2003> |
helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 21:30 |
<topranks> |
re-enable BGP for pybal on lvs1019 to fail back from lvs1020 after vlan add T440859 |
[production] |
| 21:28 |
<kemayo@deploy2003> |
mwscript-k8s job started: namespaceDupes zh_min_nanwiki --fix # T430731 |
[production] |
| 21:26 |
<cmooney@cumin1004> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host lvs1019.eqiad.wmnet |
[production] |
| 21:24 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97403 and previous config saved to /var/cache/conftool/dbconfig/20261105-212426-ladsgroup.json |
[production] |
| 21:24 |
<kemayo@deploy2003> |
Finished scap sync-world: Backport for [[gerrit:1364770|VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815|Add new namespace for zh_min_nanwiki (T430731)]] (duration: 16m 22s) |
[production] |
| 21:19 |
<cmooney@cumin1004> |
START - Cookbook sre.hosts.reboot-single for host lvs1019.eqiad.wmnet |
[production] |
| 21:17 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97402 and previous config saved to /var/cache/conftool/dbconfig/20261105-211708-ladsgroup.json |
[production] |
| 21:17 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance |
[production] |
| 21:16 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97401 and previous config saved to /var/cache/conftool/dbconfig/20261105-211647-ladsgroup.json |
[production] |
| 21:15 |
<kemayo@deploy2003> |
pppery, kemayo: Continuing with deployment |
[production] |
| 21:13 |
<kemayo@deploy2003> |
pppery, kemayo: Backport for [[gerrit:1364770|VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815|Add new namespace for zh_min_nanwiki (T430731)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 21:08 |
<kemayo@deploy2003> |
Started scap sync-world: Backport for [[gerrit:1364770|VisualEditor: Precompute edit suggestion counts on 25% of jawiki + 5% of arwiki (T442767)]], [[gerrit:1340815|Add new namespace for zh_min_nanwiki (T430731)]] |
[production] |
| 21:06 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97400 and previous config saved to /var/cache/conftool/dbconfig/20261105-210645-ladsgroup.json |
[production] |
| 20:56 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P97399 and previous config saved to /var/cache/conftool/dbconfig/20261105-205643-ladsgroup.json |
[production] |
| 20:56 |
<James_F> |
mwscript sql.php --wiki=wikifunctionswiki --cluster extension1 extensions/WikiLambda/sql/mysql/table-abstractwiki_integrations.sql # T443295 |
[production] |
| 20:46 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97398 and previous config saved to /var/cache/conftool/dbconfig/20261105-204641-ladsgroup.json |
[production] |
| 20:44 |
<topranks> |
disable pybal service on lvs1019 to failover to lvs1020 ahead of adding new vlan ints to the former T440859 |
[production] |
| 20:39 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db2166 (T441633)', diff saved to https://phabricator.wikimedia.org/P97397 and previous config saved to /var/cache/conftool/dbconfig/20261105-203938-ladsgroup.json |
[production] |
| 20:39 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance |
[production] |
| 20:39 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97396 and previous config saved to /var/cache/conftool/dbconfig/20261105-203917-ladsgroup.json |
[production] |
| 20:39 |
<cmooney@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs1019.eqiad.wmnet with reason: failover lvs1019 to lvs1020 to add new vlan sub-ints |
[production] |
| 20:29 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97395 and previous config saved to /var/cache/conftool/dbconfig/20261105-202915-ladsgroup.json |
[production] |
| 20:19 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2165', diff saved to https://phabricator.wikimedia.org/P97394 and previous config saved to /var/cache/conftool/dbconfig/20261105-201913-ladsgroup.json |
[production] |
| 20:09 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97393 and previous config saved to /var/cache/conftool/dbconfig/20261105-200911-ladsgroup.json |
[production] |
| 20:06 |
<bking@deploy2003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply |
[production] |
| 20:05 |
<bking@deploy2003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply |
[production] |
| 20:02 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db2165 (T441633)', diff saved to https://phabricator.wikimedia.org/P97392 and previous config saved to /var/cache/conftool/dbconfig/20261105-200211-ladsgroup.json |
[production] |
| 20:02 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2165.codfw.wmnet with reason: Maintenance |
[production] |
| 20:01 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97391 and previous config saved to /var/cache/conftool/dbconfig/20261105-200150-ladsgroup.json |
[production] |
| 19:56 |
<bking@deploy2003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/matomo-next: apply |
[production] |
| 19:55 |
<bking@deploy2003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/matomo-next: apply |
[production] |
| 19:51 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97390 and previous config saved to /var/cache/conftool/dbconfig/20261105-195148-ladsgroup.json |
[production] |
| 19:41 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P97389 and previous config saved to /var/cache/conftool/dbconfig/20261105-194146-ladsgroup.json |
[production] |
| 19:31 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97388 and previous config saved to /var/cache/conftool/dbconfig/20261105-193144-ladsgroup.json |
[production] |
| 19:24 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db2164 (T441633)', diff saved to https://phabricator.wikimedia.org/P97387 and previous config saved to /var/cache/conftool/dbconfig/20261105-192426-ladsgroup.json |
[production] |
| 19:24 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance |
[production] |
| 19:24 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2163 (T441633)', diff saved to https://phabricator.wikimedia.org/P97386 and previous config saved to /var/cache/conftool/dbconfig/20261105-192416-ladsgroup.json |
[production] |
| 19:14 |
<bd808@deploy2003> |
helmfile [eqiad] DONE helmfile.d/services/toolhub: apply |
[production] |
| 19:14 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97385 and previous config saved to /var/cache/conftool/dbconfig/20261105-191414-ladsgroup.json |
[production] |
| 19:13 |
<bd808@deploy2003> |
helmfile [eqiad] START helmfile.d/services/toolhub: apply |
[production] |
| 19:12 |
<bd808@deploy2003> |
helmfile [codfw] DONE helmfile.d/services/toolhub: apply |
[production] |
| 19:12 |
<bd808@deploy2003> |
helmfile [codfw] START helmfile.d/services/toolhub: apply |
[production] |
| 19:11 |
<bd808@deploy2003> |
helmfile [staging] DONE helmfile.d/services/toolhub: apply |
[production] |
| 19:10 |
<bd808@deploy2003> |
helmfile [staging] START helmfile.d/services/toolhub: apply |
[production] |
| 19:04 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P97384 and previous config saved to /var/cache/conftool/dbconfig/20261105-190412-ladsgroup.json |
[production] |