|
2026-10-22
ยง
|
| 21:03 |
<dancy@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1357836|wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064)]] (1.48.0-wmf.1) (duration: 08m 23s) |
[production] |
| 21:01 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1015.eqiad.wmnet |
[production] |
| 21:01 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1014.eqiad.wmnet |
[production] |
| 21:01 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1014.eqiad.wmnet |
[production] |
| 20:57 |
<dancy@deploy1003> |
dancy: Continuing with deployment |
[production] |
| 20:57 |
<dancy@deploy1003> |
dancy: Backport for [[gerrit:1357836|wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064)]] (1.48.0-wmf.1) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 20:55 |
<dancy@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1357836|wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064)]] (1.48.0-wmf.1) |
[production] |
| 20:53 |
<dancy@deploy1003> |
Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 11m 08s) |
[production] |
| 20:53 |
<dancy@deploy1003> |
Forcefully removing global lock: Deployments locked for the week |
[production] |
| 20:45 |
<btullis@cumin1004> |
END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) depool for host dse-k8s-worker1014.eqiad.wmnet |
[production] |
| 20:42 |
<dancy@deploy1003> |
Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week |
[production] |
| 20:42 |
<dancy@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1357790|wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064)]] (1.47.0-wmf.24) (duration: 08m 51s) |
[production] |
| 20:41 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1014.eqiad.wmnet |
[production] |
| 20:40 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1013.eqiad.wmnet |
[production] |
| 20:40 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1013.eqiad.wmnet |
[production] |
| 20:36 |
<dancy@deploy1003> |
jforrester, dancy: Continuing with deployment |
[production] |
| 20:35 |
<dancy@deploy1003> |
jforrester, dancy: Backport for [[gerrit:1357790|wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064)]] (1.47.0-wmf.24) synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 20:34 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1013.eqiad.wmnet |
[production] |
| 20:34 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1013.eqiad.wmnet |
[production] |
| 20:34 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet |
[production] |
| 20:34 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet |
[production] |
| 20:33 |
<dancy@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1357790|wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064)]] (1.47.0-wmf.24) |
[production] |
| 20:30 |
<thcipriani@deploy1003> |
Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 156m 22s) |
[production] |
| 20:30 |
<dancy@deploy1003> |
Forcefully removing global lock: Deployments locked for the week |
[production] |
| 20:29 |
<dancy@deploy1003> |
Forcefully removing global lock: Deployments locked for the week |
[production] |
| 20:28 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet |
[production] |
| 20:17 |
<rzl@cumin1004> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for phab2003.codfw.wmnet,phab1005.eqiad.wmnet |
[production] |
| 20:17 |
<rzl@cumin1004> |
START - Cookbook sre.hosts.remove-downtime for phab2003.codfw.wmnet,phab1005.eqiad.wmnet |
[production] |
| 20:10 |
<thcipriani> |
phab restart |
[production] |
| 20:10 |
<rzl@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab2003.codfw.wmnet,phab1005.eqiad.wmnet with reason: deploying |
[production] |
| 19:58 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet |
[production] |
| 19:57 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet |
[production] |
| 19:57 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet |
[production] |
| 19:51 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet |
[production] |
| 19:39 |
<brett@cumin2003> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp5017.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945) |
[production] |
| 19:39 |
<brett@cumin2003> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp5032.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945) |
[production] |
| 19:33 |
<brett@cumin2003> |
START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp5032.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945) |
[production] |
| 19:33 |
<brett@cumin2003> |
START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp5017.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945) |
[production] |
| 19:27 |
<rzl@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 19:26 |
<rzl@deploy1003> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 19:26 |
<rzl@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 19:26 |
<rzl@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 19:21 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet |
[production] |
| 19:21 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet |
[production] |
| 19:21 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet |
[production] |
| 19:15 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet |
[production] |
| 19:01 |
<rzl@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 18:59 |
<rzl@deploy1003> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:55 |
<rzl@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 18:53 |
<rzl@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |