|
2026-11-06
§
|
| 10:33 |
<jelto@deploy2003> |
scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb |
[production] |
| 10:31 |
<jelto@deploy2003> |
scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb |
[production] |
| 10:31 |
<jelto@deploy2003> |
Started scap deploy-service miscweb-aux: Deploy to fix T443357 |
[production] |
| 10:29 |
<marostegui> |
Limit icinga db user connections to 10 in ms* and pc* T442995 |
[production] |
| 10:16 |
<ayounsi@cumin1004> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox |
[production] |
| 10:16 |
<ayounsi@cumin1004> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox |
[production] |
| 10:16 |
<ayounsi@cumin1004> |
END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary |
[production] |
| 10:11 |
<fabfur@cumin1004> |
conftool action : set/pooled=yes; selector: dc=magru,cluster=dnsbox |
[production] |
| 10:08 |
<ayounsi@cumin1004> |
START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary |
[production] |
| 09:04 |
<mvernon@cumin1004> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe |
[production] |
| 08:57 |
<jelto@deploy2003> |
Finished scap deploy-service miscweb-aux: Deploy to fix T443826 (duration: 04m 11s) |
[production] |
| 08:56 |
<jelto@deploy2003> |
scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-codfw Deploying miscweb |
[production] |
| 08:53 |
<jelto@deploy2003> |
scap deploy-service miscweb-aux: [PRODUCTION/DEFAULT] aux-k8s-eqiad Deploying miscweb |
[production] |
| 08:53 |
<jelto@deploy2003> |
Started scap deploy-service miscweb-aux: Deploy to fix T443826 |
[production] |
| 08:52 |
<mvernon@cumin2003> |
END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe |
[production] |
| 08:48 |
<mvernon@cumin1004> |
START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe |
[production] |
| 08:47 |
<mvernon@cumin2003> |
START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe |
[production] |
| 06:52 |
<XioNoX> |
push pfw policies - T443852 |
[production] |
| 00:00 |
<root@deploy2003> |
helmfile [codfw] DONE helmfile.d/services/shellbox-timeline: sync |
[production] |
| 00:00 |
<root@deploy2003> |
helmfile [codfw] START helmfile.d/services/shellbox-timeline: sync |
[production] |
|
2026-11-05
§
|
| 23:30 |
<mutante> |
codesearch2001 initial repo pull from github. codesearch1001 temp puppet disabled for careful rollout. |
[production] |
| 23:18 |
<maryum> |
Deployed security fix for T443424, T443467, T443036 |
[production] |
| 23:10 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance |
[production] |
| 23:10 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97417 and previous config saved to /var/cache/conftool/dbconfig/20261105-231002-ladsgroup.json |
[production] |
| 23:00 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97416 and previous config saved to /var/cache/conftool/dbconfig/20261105-230000-ladsgroup.json |
[production] |
| 22:51 |
<sbassett> |
Deployed security fixes for T439878, T440323, T440657, T440656, T440655 |
[production] |
| 22:49 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P97415 and previous config saved to /var/cache/conftool/dbconfig/20261105-224958-ladsgroup.json |
[production] |
| 22:39 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97414 and previous config saved to /var/cache/conftool/dbconfig/20261105-223956-ladsgroup.json |
[production] |
| 22:32 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db2195 (T441633)', diff saved to https://phabricator.wikimedia.org/P97413 and previous config saved to /var/cache/conftool/dbconfig/20261105-223248-ladsgroup.json |
[production] |
| 22:32 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance |
[production] |
| 22:32 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97412 and previous config saved to /var/cache/conftool/dbconfig/20261105-223225-ladsgroup.json |
[production] |
| 22:30 |
<egardner@deploy2003> |
Finished scap sync-world: Backport for [[gerrit:1364473|ShareCard: Log client errors with the topic "error.sharecard" (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474|ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) (duration: 11m 46s) |
[production] |
| 22:24 |
<egardner@deploy2003> |
egardner: Continuing with deployment |
[production] |
| 22:23 |
<egardner@deploy2003> |
egardner: Backport for [[gerrit:1364473|ShareCard: Log client errors with the topic "error.sharecard" (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474|ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 22:22 |
<ssastry@deploy2003> |
helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply |
[production] |
| 22:22 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97411 and previous config saved to /var/cache/conftool/dbconfig/20261105-222223-ladsgroup.json |
[production] |
| 22:22 |
<ssastry@deploy2003> |
helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply |
[production] |
| 22:19 |
<egardner@deploy2003> |
Started scap sync-world: Backport for [[gerrit:1364473|ShareCard: Log client errors with the topic "error.sharecard" (T439163)]] (1.48.0-wmf.3), [[gerrit:1363474|ShareCard: Show animated GIFs as a still image (T442968)]] (1.48.0-wmf.3) |
[production] |
| 22:15 |
<egardner@deploy2003> |
Finished scap sync-world: Backport for [[gerrit:1358851|Add Creative Commons license icons for the share card (T439162)]] (duration: 14m 19s) |
[production] |
| 22:12 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P97410 and previous config saved to /var/cache/conftool/dbconfig/20261105-221221-ladsgroup.json |
[production] |
| 22:08 |
<egardner@deploy2003> |
egardner: Continuing with deployment |
[production] |
| 22:07 |
<egardner@deploy2003> |
egardner: Backport for [[gerrit:1358851|Add Creative Commons license icons for the share card (T439162)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 22:02 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97409 and previous config saved to /var/cache/conftool/dbconfig/20261105-220220-ladsgroup.json |
[production] |
| 22:01 |
<egardner@deploy2003> |
Started scap sync-world: Backport for [[gerrit:1358851|Add Creative Commons license icons for the share card (T439162)]] |
[production] |
| 21:54 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Depooling db2181 (T441633)', diff saved to https://phabricator.wikimedia.org/P97408 and previous config saved to /var/cache/conftool/dbconfig/20261105-215453-ladsgroup.json |
[production] |
| 21:54 |
<mutante> |
ci / contint1003/contint2003 - if "sudo docker info -f '{{.Driver}}'" says "overlay2" it's the classic way. after restart when it says "overlayfs2" that will be the NEW way! (T440668) |
[production] |
| 21:54 |
<ladsgroup@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance |
[production] |
| 21:54 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2167 (T441633)', diff saved to https://phabricator.wikimedia.org/P97407 and previous config saved to /var/cache/conftool/dbconfig/20261105-215432-ladsgroup.json |
[production] |
| 21:51 |
<mutante> |
ci / contint1003/contint2003 - docker will be reconfigured to use containerd-snapshotter rather than dockers legacy store, overlay2 (a thing docker has been moving towards for a while). images pulled under layer2 become invisible on next docker restart. releng will handle that and prune the old images. new images will have to be pulled down. needed for ST6.1.4 (T440668) |
[production] |
| 21:44 |
<ladsgroup@cumin1004> |
dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P97406 and previous config saved to /var/cache/conftool/dbconfig/20261105-214430-ladsgroup.json |
[production] |