|
2026-10-22
ยง
|
| 20:34 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1012.eqiad.wmnet |
[production] |
| 20:34 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1012.eqiad.wmnet |
[production] |
| 20:33 |
<dancy@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1357790|wikisource-ocr-langs: Lower timeout, cache absence, set lockTSE+busyValue (T442064)]] (1.47.0-wmf.24) |
[production] |
| 20:30 |
<thcipriani@deploy1003> |
Unlocked for deployment [ALL REPOSITORIES]: Deployments locked for the week (duration: 156m 22s) |
[production] |
| 20:30 |
<dancy@deploy1003> |
Forcefully removing global lock: Deployments locked for the week |
[production] |
| 20:29 |
<dancy@deploy1003> |
Forcefully removing global lock: Deployments locked for the week |
[production] |
| 20:28 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1012.eqiad.wmnet |
[production] |
| 20:17 |
<rzl@cumin1004> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for phab2003.codfw.wmnet,phab1005.eqiad.wmnet |
[production] |
| 20:17 |
<rzl@cumin1004> |
START - Cookbook sre.hosts.remove-downtime for phab2003.codfw.wmnet,phab1005.eqiad.wmnet |
[production] |
| 20:10 |
<thcipriani> |
phab restart |
[production] |
| 20:10 |
<rzl@cumin1004> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on phab2003.codfw.wmnet,phab1005.eqiad.wmnet with reason: deploying |
[production] |
| 19:58 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1012.eqiad.wmnet |
[production] |
| 19:57 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1011.eqiad.wmnet |
[production] |
| 19:57 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1011.eqiad.wmnet |
[production] |
| 19:51 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1011.eqiad.wmnet |
[production] |
| 19:39 |
<brett@cumin2003> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp5017.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945) |
[production] |
| 19:39 |
<brett@cumin2003> |
END (PASS) - Cookbook sre.cdn.roll-upgrade-ats (exit_code=0) Rolling upgrade of ATS on P{cp5032.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945) |
[production] |
| 19:33 |
<brett@cumin2003> |
START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp5032.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945) |
[production] |
| 19:33 |
<brett@cumin2003> |
START - Cookbook sre.cdn.roll-upgrade-ats Rolling upgrade of ATS on P{cp5017.eqsin.wmnet} and A:cp - 9.2.16 upgrade (T441945) |
[production] |
| 19:27 |
<rzl@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 19:26 |
<rzl@deploy1003> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 19:26 |
<rzl@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 19:26 |
<rzl@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 19:21 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1011.eqiad.wmnet |
[production] |
| 19:21 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1010.eqiad.wmnet |
[production] |
| 19:21 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1010.eqiad.wmnet |
[production] |
| 19:15 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1010.eqiad.wmnet |
[production] |
| 19:01 |
<rzl@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 18:59 |
<rzl@deploy1003> |
helmfile [codfw] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:55 |
<rzl@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply |
[production] |
| 18:53 |
<rzl@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-debug: apply |
[production] |
| 18:45 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1010.eqiad.wmnet |
[production] |
| 18:45 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1009.eqiad.wmnet |
[production] |
| 18:45 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1009.eqiad.wmnet |
[production] |
| 18:39 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1009.eqiad.wmnet |
[production] |
| 18:09 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1009.eqiad.wmnet |
[production] |
| 18:08 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1008.eqiad.wmnet |
[production] |
| 18:08 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1008.eqiad.wmnet |
[production] |
| 18:01 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1008.eqiad.wmnet |
[production] |
| 18:01 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1008.eqiad.wmnet |
[production] |
| 18:01 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1007.eqiad.wmnet |
[production] |
| 18:00 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1007.eqiad.wmnet |
[production] |
| 17:54 |
<mutante> |
codesearch zombie hound procs piling up. codesearch1001 ran out of PIDs. |
[production] |
| 17:54 |
<thcipriani@deploy1003> |
Locking from deployment [ALL REPOSITORIES]: Deployments locked for the week |
[production] |
| 17:53 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host dse-k8s-worker1007.eqiad.wmnet |
[production] |
| 17:53 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node depool for host dse-k8s-worker1007.eqiad.wmnet |
[production] |
| 17:53 |
<btullis@cumin1004> |
END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host dse-k8s-worker1006.eqiad.wmnet |
[production] |
| 17:53 |
<btullis@cumin1004> |
START - Cookbook sre.k8s.pool-depool-node pool for host dse-k8s-worker1006.eqiad.wmnet |
[production] |
| 17:50 |
<ladsgroup@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1357769|Upgrading composer/ca-bundle (1.5.14 => 1.6.0)]] (1.47.0-wmf.24) (duration: 43m 35s) |
[production] |
| 17:46 |
<mutante> |
codesearch1001 - hard rebooting at ganeti level |
[production] |