|
2025-10-17
§
|
| 12:07 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db1195 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P84062 and previous config saved to /var/cache/conftool/dbconfig/20251017-120737-marostegui.json |
[production] |
| 12:07 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1195.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:38 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db2248.codfw.wmnet onto db2246.codfw.wmnet |
[production] |
| 11:38 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.pool (exit_code=0) db2248 gradually with 4 steps - Pool db2248.codfw.wmnet in after cloning |
[production] |
| 11:11 |
<cgoubert@deploy2002> |
helmfile [codfw] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:11 |
<cgoubert@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:11 |
<cgoubert@deploy2002> |
helmfile [eqiad] START helmfile.d/services/rest-gateway: apply |
[production] |
| 11:06 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 11:06 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:52 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.pool db2248 gradually with 4 steps - Pool db2248.codfw.wmnet in after cloning |
[production] |
| 10:44 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 10:43 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:36 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 10:35 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:35 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 10:34 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/rest-gateway: apply |
[production] |
| 10:08 |
<eileen> |
civicrm upgraded from ab1d21dc to 7b70cb83 |
[production] |
| 10:05 |
<klausman@deploy2002> |
helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:05 |
<klausman@deploy2002> |
helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 10:03 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:03 |
<topranks> |
un-draining Arelion 100G transport eqiad <-> codfw following carrier fibre fix and return to stability T407578 |
[production] |
| 10:03 |
<klausman@deploy2002> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 10:02 |
<klausman@deploy2002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 10:02 |
<klausman@deploy2002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 09:37 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 09:36 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/rest-gateway: apply |
[production] |
| 08:47 |
<cgoubert@deploy2002> |
helmfile [staging] DONE helmfile.d/services/rest-gateway: apply |
[production] |
| 08:46 |
<cgoubert@deploy2002> |
helmfile [staging] START helmfile.d/services/rest-gateway: apply |
[production] |
| 08:19 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.mysql.depool (exit_code=0) db2248 - Depool db2248.codfw.wmnet to then clone it to db2246.codfw.wmnet - marostegui@cumin1003 |
[production] |
| 08:19 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.depool db2248 - Depool db2248.codfw.wmnet to then clone it to db2246.codfw.wmnet - marostegui@cumin1003 |
[production] |
| 08:19 |
<marostegui@cumin1003> |
START - Cookbook sre.mysql.clone of db2248.codfw.wmnet onto db2246.codfw.wmnet |
[production] |
| 08:08 |
<topranks> |
draining Arelion eqiad <-> codfw transport wiht OSPF metric and re-enabling port on cr1-eqiad |
[production] |
| 08:04 |
<fceratto@cumin1003> |
START - Cookbook sre.mysql.clone_es of es2032.codfw.wmnet onto es2055.codfw.wmnet |
[production] |
| 07:42 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1056 (re)pooling @ 100%: Host provisioned T406488', diff saved to https://phabricator.wikimedia.org/P84056 and previous config saved to /var/cache/conftool/dbconfig/20251017-074221-root.json |
[production] |
| 07:27 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1056 (re)pooling @ 75%: Host provisioned T406488', diff saved to https://phabricator.wikimedia.org/P84055 and previous config saved to /var/cache/conftool/dbconfig/20251017-072715-root.json |
[production] |
| 07:12 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1056 (re)pooling @ 60%: Host provisioned T406488', diff saved to https://phabricator.wikimedia.org/P84054 and previous config saved to /var/cache/conftool/dbconfig/20251017-071209-root.json |
[production] |
| 06:57 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1056 (re)pooling @ 50%: Host provisioned T406488', diff saved to https://phabricator.wikimedia.org/P84053 and previous config saved to /var/cache/conftool/dbconfig/20251017-065703-root.json |
[production] |
| 06:41 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1056 (re)pooling @ 30%: Host provisioned T406488', diff saved to https://phabricator.wikimedia.org/P84052 and previous config saved to /var/cache/conftool/dbconfig/20251017-064157-root.json |
[production] |
| 06:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1056 (re)pooling @ 25%: Host provisioned T406488', diff saved to https://phabricator.wikimedia.org/P84051 and previous config saved to /var/cache/conftool/dbconfig/20251017-062651-root.json |
[production] |
| 06:11 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1056 (re)pooling @ 20%: Host provisioned T406488', diff saved to https://phabricator.wikimedia.org/P84050 and previous config saved to /var/cache/conftool/dbconfig/20251017-061145-root.json |
[production] |
| 05:56 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1056 (re)pooling @ 10%: Host provisioned T406488', diff saved to https://phabricator.wikimedia.org/P84049 and previous config saved to /var/cache/conftool/dbconfig/20251017-055639-root.json |
[production] |
| 05:45 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on es1027.eqiad.wmnet with reason: Cloning |
[production] |
| 05:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool es1027 T407595', diff saved to https://phabricator.wikimedia.org/P84048 and previous config saved to /var/cache/conftool/dbconfig/20251017-054458-marostegui.json |
[production] |
| 05:41 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1056 (re)pooling @ 7%: Host provisioned T406488', diff saved to https://phabricator.wikimedia.org/P84047 and previous config saved to /var/cache/conftool/dbconfig/20251017-054133-root.json |
[production] |
| 05:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1056 (re)pooling @ 5%: Host provisioned T406488', diff saved to https://phabricator.wikimedia.org/P84046 and previous config saved to /var/cache/conftool/dbconfig/20251017-052627-root.json |
[production] |
| 05:11 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1056 (re)pooling @ 1%: Host provisioned T406488', diff saved to https://phabricator.wikimedia.org/P84045 and previous config saved to /var/cache/conftool/dbconfig/20251017-051121-root.json |
[production] |
| 05:11 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Add es1056 to dbctl T406488', diff saved to https://phabricator.wikimedia.org/P84044 and previous config saved to /var/cache/conftool/dbconfig/20251017-051114-marostegui.json |
[production] |
| 01:15 |
<mwpresync@deploy2002> |
Finished scap build-images: Publishing wmf/next image (duration: 14m 04s) |
[production] |
| 01:01 |
<mwpresync@deploy2002> |
Started scap build-images: Publishing wmf/next image |
[production] |