|
2025-05-27
§
|
| 06:58 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti-test2002.codfw.wmnet |
[production] |
| 06:38 |
<moritzm> |
failover Ganeti master in magru/B4 to ganeti7004 T394263 |
[production] |
| 06:31 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti-test2002.codfw.wmnet |
[production] |
| 04:01 |
<mwpresync@deploy1003> |
Pruned MediaWiki: 1.44.0-wmf.28 (duration: 01m 50s) |
[production] |
| 03:49 |
<mwpresync@deploy1003> |
Finished scap sync-world: testwikis to 1.45.0-wmf.3 refs T392173 (duration: 46m 49s) |
[production] |
| 03:02 |
<mwpresync@deploy1003> |
Started scap sync-world: testwikis to 1.45.0-wmf.3 refs T392173 |
[production] |
| 00:20 |
<dani@deploy1003> |
helmfile [codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 00:20 |
<dani@deploy1003> |
helmfile [codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 00:20 |
<dani@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/miscweb: apply |
[production] |
| 00:20 |
<dani@deploy1003> |
helmfile [eqiad] START helmfile.d/services/miscweb: apply |
[production] |
| 00:20 |
<dani@deploy1003> |
helmfile [staging] DONE helmfile.d/services/miscweb: apply |
[production] |
| 00:20 |
<dani@deploy1003> |
helmfile [staging] START helmfile.d/services/miscweb: apply |
[production] |
|
2025-05-26
§
|
| 21:56 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2228 (T395241)', diff saved to https://phabricator.wikimedia.org/P76461 and previous config saved to /var/cache/conftool/dbconfig/20250526-215649-fceratto.json |
[production] |
| 21:41 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P76460 and previous config saved to /var/cache/conftool/dbconfig/20250526-214142-fceratto.json |
[production] |
| 21:26 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2228', diff saved to https://phabricator.wikimedia.org/P76459 and previous config saved to /var/cache/conftool/dbconfig/20250526-212634-fceratto.json |
[production] |
| 21:11 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2228 (T395241)', diff saved to https://phabricator.wikimedia.org/P76458 and previous config saved to /var/cache/conftool/dbconfig/20250526-211127-fceratto.json |
[production] |
| 21:04 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db2228 (T395241)', diff saved to https://phabricator.wikimedia.org/P76457 and previous config saved to /var/cache/conftool/dbconfig/20250526-210445-fceratto.json |
[production] |
| 21:04 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance |
[production] |
| 21:04 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2228.codfw.wmnet with reason: Maintenance |
[production] |
| 21:04 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2223 (T395241)', diff saved to https://phabricator.wikimedia.org/P76456 and previous config saved to /var/cache/conftool/dbconfig/20250526-210402-fceratto.json |
[production] |
| 20:48 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P76455 and previous config saved to /var/cache/conftool/dbconfig/20250526-204855-fceratto.json |
[production] |
| 20:33 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2223', diff saved to https://phabricator.wikimedia.org/P76454 and previous config saved to /var/cache/conftool/dbconfig/20250526-203348-fceratto.json |
[production] |
| 20:18 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2223 (T395241)', diff saved to https://phabricator.wikimedia.org/P76453 and previous config saved to /var/cache/conftool/dbconfig/20250526-201840-fceratto.json |
[production] |
| 20:11 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db2223 (T395241)', diff saved to https://phabricator.wikimedia.org/P76452 and previous config saved to /var/cache/conftool/dbconfig/20250526-201150-fceratto.json |
[production] |
| 20:11 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2223.codfw.wmnet with reason: Maintenance |
[production] |
| 20:11 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2211 (T395241)', diff saved to https://phabricator.wikimedia.org/P76451 and previous config saved to /var/cache/conftool/dbconfig/20250526-201123-fceratto.json |
[production] |
| 19:56 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P76450 and previous config saved to /var/cache/conftool/dbconfig/20250526-195614-fceratto.json |
[production] |
| 19:41 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P76449 and previous config saved to /var/cache/conftool/dbconfig/20250526-194107-fceratto.json |
[production] |
| 19:33 |
<aokoth@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on doc1004.eqiad.wmnet with reason: Bookworm |
[production] |
| 19:30 |
<denisse> |
Re-enable sync between grafana hosts - T395098 |
[production] |
| 19:29 |
<aokoth@deploy1003> |
helmfile [aux-k8s-codfw] DONE helmfile.d/services/miscweb: apply |
[production] |
| 19:28 |
<aokoth@deploy1003> |
helmfile [aux-k8s-codfw] START helmfile.d/services/miscweb: apply |
[production] |
| 19:26 |
<aokoth@deploy1003> |
helmfile [aux-k8s-codfw] DONE helmfile.d/admin 'apply'. |
[production] |
| 19:26 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2211 (T395241)', diff saved to https://phabricator.wikimedia.org/P76448 and previous config saved to /var/cache/conftool/dbconfig/20250526-192600-fceratto.json |
[production] |
| 19:24 |
<aokoth@deploy1003> |
helmfile [aux-k8s-codfw] START helmfile.d/admin 'apply'. |
[production] |
| 19:19 |
<denisse> |
Upgrading Grafana to v12.0.1 on grafana1002 - T395098 |
[production] |
| 19:19 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db2211 (T395241)', diff saved to https://phabricator.wikimedia.org/P76447 and previous config saved to /var/cache/conftool/dbconfig/20250526-191912-fceratto.json |
[production] |
| 19:19 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2211.codfw.wmnet with reason: Maintenance |
[production] |
| 19:16 |
<denisse> |
Add Grafana v12.0.1 to reprepro for bookworm - T395098 |
[production] |
| 19:14 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2201.codfw.wmnet with reason: Maintenance |
[production] |
| 19:13 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192 (T395241)', diff saved to https://phabricator.wikimedia.org/P76446 and previous config saved to /var/cache/conftool/dbconfig/20250526-191341-fceratto.json |
[production] |
| 18:58 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P76445 and previous config saved to /var/cache/conftool/dbconfig/20250526-185832-fceratto.json |
[production] |
| 18:43 |
<aokoth@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on doc1003.eqiad.wmnet with reason: Bookworm |
[production] |
| 18:43 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P76444 and previous config saved to /var/cache/conftool/dbconfig/20250526-184325-fceratto.json |
[production] |
| 18:28 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2192 (T395241)', diff saved to https://phabricator.wikimedia.org/P76443 and previous config saved to /var/cache/conftool/dbconfig/20250526-182817-fceratto.json |
[production] |
| 18:22 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db2192 (T395241)', diff saved to https://phabricator.wikimedia.org/P76442 and previous config saved to /var/cache/conftool/dbconfig/20250526-182247-fceratto.json |
[production] |
| 18:22 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2192.codfw.wmnet with reason: Maintenance |
[production] |
| 18:22 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178 (T395241)', diff saved to https://phabricator.wikimedia.org/P76441 and previous config saved to /var/cache/conftool/dbconfig/20250526-182221-fceratto.json |
[production] |
| 18:07 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P76440 and previous config saved to /var/cache/conftool/dbconfig/20250526-180714-fceratto.json |
[production] |
| 17:52 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P76439 and previous config saved to /var/cache/conftool/dbconfig/20250526-175207-fceratto.json |
[production] |