|
2025-04-15
ยง
|
| 10:38 |
<sukhe> |
sudo cumin 'A:durum' 'disable-puppet "rolling out CR 1132669"' |
[production] |
| 10:37 |
<vgutierrez@cumin1002> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=0) rolling upgrade of Varnish on A:cp-upload_codfw |
[production] |
| 10:34 |
<ladsgroup@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1136670|Bump thumbnail steps to 95% (T360589)]] |
[production] |
| 10:33 |
<ladsgroup@deploy1003> |
sync-world aborted: Backport for [[gerrit:1136670|Bump thumbnail steps to 95% (T360589)]] (duration: 14m 11s) |
[production] |
| 10:27 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P75008 and previous config saved to /var/cache/conftool/dbconfig/20250415-102705-fceratto.json |
[production] |
| 10:26 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on P{cp[5023-5024].eqsin.wmnet} and A:cp |
[production] |
| 10:24 |
<vgutierrez@cumin1002> |
END (FAIL) - Cookbook sre.cdn.roll-upgrade-varnish (exit_code=1) rolling upgrade of Varnish on A:cp-text_eqsin |
[production] |
| 10:19 |
<ladsgroup@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1136670|Bump thumbnail steps to 95% (T360589)]] |
[production] |
| 10:11 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P75007 and previous config saved to /var/cache/conftool/dbconfig/20250415-101158-fceratto.json |
[production] |
| 10:00 |
<dcausse@deploy1003> |
Finished deploy [wdqs/wdqs@fe88851] (wcqs): version 0.3.156 (duration: 02m 25s) |
[production] |
| 09:58 |
<dcausse@deploy1003> |
Started deploy [wdqs/wdqs@fe88851] (wcqs): version 0.3.156 |
[production] |
| 09:57 |
<dcausse@deploy1003> |
Finished deploy [wdqs/wdqs@fe88851]: version 0.3.156 (T326311) (duration: 14m 31s) |
[production] |
| 09:56 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T391056)', diff saved to https://phabricator.wikimedia.org/P75006 and previous config saved to /var/cache/conftool/dbconfig/20250415-095650-fceratto.json |
[production] |
| 09:54 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1158 (T391056)', diff saved to https://phabricator.wikimedia.org/P75005 and previous config saved to /var/cache/conftool/dbconfig/20250415-095442-fceratto.json |
[production] |
| 09:54 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1014,1018].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:54 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1158.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:43 |
<dcausse@deploy1003> |
Started deploy [wdqs/wdqs@fe88851]: version 0.3.156 (T326311) |
[production] |
| 09:15 |
<jnuche@deploy1003> |
sync-world aborted: testwikis to 1.44.0-wmf.25 refs T386220 (duration: 14m 36s) |
[production] |
| 09:00 |
<jnuche@deploy1003> |
Started scap sync-world: testwikis to 1.44.0-wmf.25 refs T386220 |
[production] |
| 08:51 |
<dcausse@deploy1003> |
Finished deploy [wdqs/wdqs@4186ae7] (wcqs): test deploy new scap config to wcqs2001.codfw.wmnet (T221709) (duration: 00m 20s) |
[production] |
| 08:51 |
<dcausse@deploy1003> |
Started deploy [wdqs/wdqs@4186ae7] (wcqs): test deploy new scap config to wcqs2001.codfw.wmnet (T221709) |
[production] |
| 08:42 |
<XioNoX> |
drain arelion eqsin-codfw link |
[production] |
| 08:09 |
<dcausse@deploy1003> |
Finished deploy [wdqs/wdqs@4186ae7]: test deploy new scap config to wdqs2025.codfw.wmnet (T221709) (duration: 00m 18s) |
[production] |
| 08:09 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-test-k8s: apply |
[production] |
| 08:09 |
<dcausse@deploy1003> |
Started deploy [wdqs/wdqs@4186ae7]: test deploy new scap config to wdqs2025.codfw.wmnet (T221709) |
[production] |
| 08:08 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-test-k8s: apply |
[production] |
| 07:47 |
<godog> |
upgrade thanos to 0.38.0 on prometheus100[57] - T383966 |
[production] |
| 07:28 |
<Emperor> |
make sure all disks are mounted correctly prior to disk-swap testing T391854 ms-be1091 |
[production] |
| 07:28 |
<Emperor> |
make sure all disks are mounted correctly prior to disk-swap testing T391854 |
[production] |
| 07:10 |
<elukey@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on ms-be1091.eqiad.wmnet with reason: dcops maintenance |
[production] |
| 07:06 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_codfw |
[production] |
| 07:06 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_codfw |
[production] |
| 07:06 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-upload_eqsin |
[production] |
| 07:05 |
<kartik@deploy1003> |
helmfile [staging] DONE helmfile.d/services/machinetranslation: apply |
[production] |
| 07:05 |
<vgutierrez@cumin1002> |
START - Cookbook sre.cdn.roll-upgrade-varnish rolling upgrade of Varnish on A:cp-text_eqsin |
[production] |
| 07:04 |
<vgutierrez> |
rolling upgrade to varnish 7.1.1-1.1~bpo11+wmf3 in eqsin and codfw - T391334 |
[production] |
| 06:50 |
<kartik@deploy1003> |
helmfile [staging] START helmfile.d/services/machinetranslation: apply |
[production] |
| 06:48 |
<kart_> |
Updated cxserver to 2025-04-07-053106-production (T390732, T390711) |
[production] |
| 06:48 |
<kartik@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/cxserver: apply |
[production] |
| 06:47 |
<kartik@deploy1003> |
helmfile [eqiad] START helmfile.d/services/cxserver: apply |
[production] |
| 06:46 |
<kartik@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/cxserver: apply |
[production] |
| 06:45 |
<kartik@deploy1003> |
helmfile [codfw] START helmfile.d/services/cxserver: apply |
[production] |
| 06:45 |
<kartik@deploy1003> |
helmfile [staging] DONE helmfile.d/services/cxserver: apply |
[production] |
| 06:44 |
<kartik@deploy1003> |
helmfile [staging] START helmfile.d/services/cxserver: apply |
[production] |
| 05:03 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Repool pc6 T391454', diff saved to https://phabricator.wikimedia.org/P75003 and previous config saved to /var/cache/conftool/dbconfig/20250415-050307-marostegui.json |
[production] |
| 04:57 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on pc2016.codfw.wmnet,pc1016.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:57 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Depool pc6 T391454', diff saved to https://phabricator.wikimedia.org/P75002 and previous config saved to /var/cache/conftool/dbconfig/20250415-045700-marostegui.json |
[production] |
| 04:10 |
<mwpresync@deploy1003> |
Pruned MediaWiki: 1.44.0-wmf.22 (duration: 10m 03s) |
[production] |
| 03:43 |
<mwpresync@deploy1003> |
sync-world failed: <CalledProcessError> Command 'sudo -u mwbuilder /srv/mwbuilder/release/make-container-image/build-images.py /srv/mediawiki-staging/scap/image-build --staging-dir /srv/mediawiki-staging --mediawiki-versions 1.44.0-wmf.24,1.44.0-wmf.25 --multiversion-image-name docker-registry.discovery.wmnet/restricted/mediawiki-multiversion --multiversion-debug-image-name docker-registry.discov |
[production] |
| 03:02 |
<mwpresync@deploy1003> |
Started scap sync-world: testwikis to 1.44.0-wmf.25 refs T386220 |
[production] |