|
2025-05-06
ยง
|
| 11:37 |
<jynus@cumin1002> |
DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on backup[2010-2014].codfw.wmnet with reason: Upgrade and restart |
[production] |
| 11:36 |
<jynus@cumin1002> |
DONE (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 6:00:00 on backup1013.eqiad.wmnet with reason: Upgrade and restart |
[production] |
| 11:30 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P75766 and previous config saved to /var/cache/conftool/dbconfig/20250506-113031-ladsgroup.json |
[production] |
| 11:15 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1186 (T382778)', diff saved to https://phabricator.wikimedia.org/P75765 and previous config saved to /var/cache/conftool/dbconfig/20250506-111524-ladsgroup.json |
[production] |
| 11:11 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1186 (T382778)', diff saved to https://phabricator.wikimedia.org/P75764 and previous config saved to /var/cache/conftool/dbconfig/20250506-111157-ladsgroup.json |
[production] |
| 11:11 |
<ladsgroup@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1186.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:11 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T382778)', diff saved to https://phabricator.wikimedia.org/P75763 and previous config saved to /var/cache/conftool/dbconfig/20250506-111146-ladsgroup.json |
[production] |
| 10:56 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P75762 and previous config saved to /var/cache/conftool/dbconfig/20250506-105639-ladsgroup.json |
[production] |
| 10:41 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P75761 and previous config saved to /var/cache/conftool/dbconfig/20250506-104131-ladsgroup.json |
[production] |
| 10:26 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1184 (T382778)', diff saved to https://phabricator.wikimedia.org/P75760 and previous config saved to /var/cache/conftool/dbconfig/20250506-102624-ladsgroup.json |
[production] |
| 10:22 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1184 (T382778)', diff saved to https://phabricator.wikimedia.org/P75759 and previous config saved to /var/cache/conftool/dbconfig/20250506-102236-ladsgroup.json |
[production] |
| 10:22 |
<ladsgroup@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1184.eqiad.wmnet with reason: Maintenance |
[production] |
| 10:22 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169 (T382778)', diff saved to https://phabricator.wikimedia.org/P75758 and previous config saved to /var/cache/conftool/dbconfig/20250506-102226-ladsgroup.json |
[production] |
| 10:07 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P75757 and previous config saved to /var/cache/conftool/dbconfig/20250506-100719-ladsgroup.json |
[production] |
| 09:57 |
<jgiannelos@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 09:57 |
<jgiannelos@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mobileapps: apply |
[production] |
| 09:56 |
<jgiannelos@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 09:56 |
<jgiannelos@deploy1003> |
helmfile [codfw] START helmfile.d/services/mobileapps: apply |
[production] |
| 09:56 |
<jgiannelos@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 09:56 |
<jgiannelos@deploy1003> |
helmfile [codfw] START helmfile.d/services/mobileapps: apply |
[production] |
| 09:55 |
<jgiannelos@deploy1003> |
helmfile [staging] DONE helmfile.d/services/mobileapps: apply |
[production] |
| 09:55 |
<jgiannelos@deploy1003> |
helmfile [staging] START helmfile.d/services/mobileapps: apply |
[production] |
| 09:52 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P75756 and previous config saved to /var/cache/conftool/dbconfig/20250506-095212-ladsgroup.json |
[production] |
| 09:44 |
<jgiannelos@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/changeprop: apply |
[production] |
| 09:43 |
<jgiannelos@deploy1003> |
helmfile [eqiad] START helmfile.d/services/changeprop: apply |
[production] |
| 09:42 |
<jgiannelos@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/changeprop: apply |
[production] |
| 09:42 |
<jgiannelos@deploy1003> |
helmfile [codfw] START helmfile.d/services/changeprop: apply |
[production] |
| 09:41 |
<jgiannelos@deploy1003> |
helmfile [staging] DONE helmfile.d/services/changeprop: apply |
[production] |
| 09:40 |
<jgiannelos@deploy1003> |
helmfile [staging] START helmfile.d/services/changeprop: apply |
[production] |
| 09:40 |
<jgiannelos@deploy1003> |
helmfile [staging] DONE helmfile.d/services/changeprop: apply |
[production] |
| 09:40 |
<jgiannelos@deploy1003> |
helmfile [staging] START helmfile.d/services/changeprop: apply |
[production] |
| 09:40 |
<fnegri@cumin1002> |
END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database nupwiki (T390714) |
[production] |
| 09:40 |
<fnegri@cumin1002> |
START - Cookbook sre.wikireplicas.add-wiki for database nupwiki (T390714) |
[production] |
| 09:37 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1169 (T382778)', diff saved to https://phabricator.wikimedia.org/P75755 and previous config saved to /var/cache/conftool/dbconfig/20250506-093704-ladsgroup.json |
[production] |
| 09:34 |
<ladsgroup@cumin1002> |
dbctl commit (dc=all): 'Depooling db1169 (T382778)', diff saved to https://phabricator.wikimedia.org/P75754 and previous config saved to /var/cache/conftool/dbconfig/20250506-093410-ladsgroup.json |
[production] |
| 09:34 |
<ladsgroup@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1169.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:28 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply |
[production] |
| 09:28 |
<lucaswerkmeister-wmde@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/wikidata-query-gui: apply |
[production] |
| 09:28 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply |
[production] |
| 09:28 |
<lucaswerkmeister-wmde@deploy1003> |
helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply |
[production] |
| 09:28 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply |
[production] |
| 09:27 |
<lucaswerkmeister-wmde@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply |
[production] |
| 09:27 |
<brouberol@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply |
[production] |
| 09:27 |
<lucaswerkmeister-wmde@deploy1003> |
helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply |
[production] |
| 09:26 |
<lucaswerkmeister-wmde@deploy1003> |
helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply |
[production] |
| 09:26 |
<lucaswerkmeister-wmde@deploy1003> |
helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply |
[production] |
| 07:49 |
<elukey> |
restart apache2 on puppetmaster1001 |
[production] |
| 04:07 |
<mwpresync@deploy1003> |
Pruned MediaWiki: 1.44.0-wmf.24 (duration: 07m 35s) |
[production] |
| 04:06 |
<mwpresync@deploy1003> |
Finished scap sync-world: testwikis to 1.44.0-wmf.28 refs T386223 (duration: 62m 44s) |
[production] |
| 03:52 |
<ryankemper@cumin2002> |
END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards |
[production] |