|
2025-10-31
ยง
|
| 09:14 |
<mvernon@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on ms-be1090.eqiad.wmnet with reason: host reimage |
[production] |
| 09:12 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2163 (T407997)', diff saved to https://phabricator.wikimedia.org/P84535 and previous config saved to /var/cache/conftool/dbconfig/20251031-091230-marostegui.json |
[production] |
| 09:01 |
<mvernon@cumin1003> |
START - Cookbook sre.hosts.reimage for host ms-be1090.eqiad.wmnet with OS bullseye |
[production] |
| 08:59 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1214 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P84534 and previous config saved to /var/cache/conftool/dbconfig/20251031-085934-root.json |
[production] |
| 08:58 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2163 (T407997)', diff saved to https://phabricator.wikimedia.org/P84533 and previous config saved to /var/cache/conftool/dbconfig/20251031-085852-marostegui.json |
[production] |
| 08:58 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2163.codfw.wmnet with reason: Maintenance |
[production] |
| 08:58 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2161 (T407997)', diff saved to https://phabricator.wikimedia.org/P84532 and previous config saved to /var/cache/conftool/dbconfig/20251031-085827-marostegui.json |
[production] |
| 08:54 |
<elukey@cumin2002> |
END (PASS) - Cookbook sre.hosts.powercycle (exit_code=0) for host sretest2010 |
[production] |
| 08:52 |
<elukey@cumin2002> |
START - Cookbook sre.hosts.powercycle for host sretest2010 |
[production] |
| 08:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1214 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P84531 and previous config saved to /var/cache/conftool/dbconfig/20251031-084428-root.json |
[production] |
| 08:43 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P84530 and previous config saved to /var/cache/conftool/dbconfig/20251031-084320-marostegui.json |
[production] |
| 08:29 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1214 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P84529 and previous config saved to /var/cache/conftool/dbconfig/20251031-082923-root.json |
[production] |
| 08:28 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P84528 and previous config saved to /var/cache/conftool/dbconfig/20251031-082812-marostegui.json |
[production] |
| 08:14 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1214 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P84527 and previous config saved to /var/cache/conftool/dbconfig/20251031-081417-root.json |
[production] |
| 08:13 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2161 (T407997)', diff saved to https://phabricator.wikimedia.org/P84526 and previous config saved to /var/cache/conftool/dbconfig/20251031-081304-marostegui.json |
[production] |
| 08:06 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db1214 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P84525 and previous config saved to /var/cache/conftool/dbconfig/20251031-080633-marostegui.json |
[production] |
| 08:06 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1214.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:59 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2161 (T407997)', diff saved to https://phabricator.wikimedia.org/P84524 and previous config saved to /var/cache/conftool/dbconfig/20251031-075931-marostegui.json |
[production] |
| 07:59 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2161.codfw.wmnet with reason: Maintenance |
[production] |
| 07:59 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2154 (T407997)', diff saved to https://phabricator.wikimedia.org/P84523 and previous config saved to /var/cache/conftool/dbconfig/20251031-075907-marostegui.json |
[production] |
| 07:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P84522 and previous config saved to /var/cache/conftool/dbconfig/20251031-074359-marostegui.json |
[production] |
| 07:28 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P84521 and previous config saved to /var/cache/conftool/dbconfig/20251031-072852-marostegui.json |
[production] |
| 07:13 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2154 (T407997)', diff saved to https://phabricator.wikimedia.org/P84520 and previous config saved to /var/cache/conftool/dbconfig/20251031-071344-marostegui.json |
[production] |
| 07:12 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2173 (re)pooling @ 100%: After upgrading', diff saved to https://phabricator.wikimedia.org/P84519 and previous config saved to /var/cache/conftool/dbconfig/20251031-071243-root.json |
[production] |
| 07:04 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1226 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P84518 and previous config saved to /var/cache/conftool/dbconfig/20251031-070422-root.json |
[production] |
| 06:59 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2154 (T407997)', diff saved to https://phabricator.wikimedia.org/P84517 and previous config saved to /var/cache/conftool/dbconfig/20251031-065953-marostegui.json |
[production] |
| 06:59 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2154.codfw.wmnet with reason: Maintenance |
[production] |
| 06:59 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2152 (T407997)', diff saved to https://phabricator.wikimedia.org/P84516 and previous config saved to /var/cache/conftool/dbconfig/20251031-065929-marostegui.json |
[production] |
| 06:57 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2173 (re)pooling @ 75%: After upgrading', diff saved to https://phabricator.wikimedia.org/P84515 and previous config saved to /var/cache/conftool/dbconfig/20251031-065737-root.json |
[production] |
| 06:49 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1226 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P84514 and previous config saved to /var/cache/conftool/dbconfig/20251031-064916-root.json |
[production] |
| 06:44 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P84513 and previous config saved to /var/cache/conftool/dbconfig/20251031-064422-marostegui.json |
[production] |
| 06:42 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2173 (re)pooling @ 50%: After upgrading', diff saved to https://phabricator.wikimedia.org/P84512 and previous config saved to /var/cache/conftool/dbconfig/20251031-064231-root.json |
[production] |
| 06:34 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1226 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P84511 and previous config saved to /var/cache/conftool/dbconfig/20251031-063410-root.json |
[production] |
| 06:29 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P84510 and previous config saved to /var/cache/conftool/dbconfig/20251031-062914-marostegui.json |
[production] |
| 06:27 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2173 (re)pooling @ 25%: After upgrading', diff saved to https://phabricator.wikimedia.org/P84509 and previous config saved to /var/cache/conftool/dbconfig/20251031-062725-root.json |
[production] |
| 06:19 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1226 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P84508 and previous config saved to /var/cache/conftool/dbconfig/20251031-061904-root.json |
[production] |
| 06:14 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2152 (T407997)', diff saved to https://phabricator.wikimedia.org/P84507 and previous config saved to /var/cache/conftool/dbconfig/20251031-061406-marostegui.json |
[production] |
| 06:12 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2173 (re)pooling @ 10%: After upgrading', diff saved to https://phabricator.wikimedia.org/P84506 and previous config saved to /var/cache/conftool/dbconfig/20251031-061219-root.json |
[production] |
| 06:11 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db1226 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P84505 and previous config saved to /var/cache/conftool/dbconfig/20251031-061110-marostegui.json |
[production] |
| 06:11 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1226.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:04 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2173 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P84504 and previous config saved to /var/cache/conftool/dbconfig/20251031-060405-marostegui.json |
[production] |
| 06:04 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2173.codfw.wmnet with reason: Maintenance |
[production] |
| 06:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2152 (T407997)', diff saved to https://phabricator.wikimedia.org/P84503 and previous config saved to /var/cache/conftool/dbconfig/20251031-060012-marostegui.json |
[production] |
| 06:00 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2152.codfw.wmnet with reason: Maintenance |
[production] |
| 05:47 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1193.eqiad.wmnet with reason: Maintenance |
[production] |
| 03:21 |
<ejegg> |
restarted donations queue consumer |
[production] |
| 03:01 |
<ejegg> |
temporarily disabled donations queue consumer to get Acoustic export to work |
[production] |
| 02:48 |
<tstarling@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1199892|Enable ChangesListQuery partitioning on all wikis (T403798)]] (duration: 40m 01s) |
[production] |
| 02:43 |
<tstarling@deploy2002> |
tstarling: Continuing with sync |
[production] |
| 02:12 |
<tstarling@deploy2002> |
tstarling: Backport for [[gerrit:1199892|Enable ChangesListQuery partitioning on all wikis (T403798)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |