|
2026-02-04
ยง
|
| 07:35 |
<marostegui> |
Deploy schema change on db2204 (old s2 codfw master) T415786 |
[production] |
| 07:34 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on db2204.codfw.wmnet with reason: Schema change |
[production] |
| 07:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1253 (T415786)', diff saved to https://phabricator.wikimedia.org/P88623 and previous config saved to /var/cache/conftool/dbconfig/20260204-072658-marostegui.json |
[production] |
| 07:26 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1253.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1231 (T415786)', diff saved to https://phabricator.wikimedia.org/P88622 and previous config saved to /var/cache/conftool/dbconfig/20260204-072632-marostegui.json |
[production] |
| 07:11 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P88621 and previous config saved to /var/cache/conftool/dbconfig/20260204-071124-marostegui.json |
[production] |
| 06:56 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P88620 and previous config saved to /var/cache/conftool/dbconfig/20260204-065616-marostegui.json |
[production] |
| 06:41 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1239.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:41 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1235 (T415786)', diff saved to https://phabricator.wikimedia.org/P88619 and previous config saved to /var/cache/conftool/dbconfig/20260204-064118-marostegui.json |
[production] |
| 06:41 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1231 (T415786)', diff saved to https://phabricator.wikimedia.org/P88618 and previous config saved to /var/cache/conftool/dbconfig/20260204-064107-marostegui.json |
[production] |
| 06:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P88617 and previous config saved to /var/cache/conftool/dbconfig/20260204-063103-marostegui.json |
[production] |
| 06:20 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P88616 and previous config saved to /var/cache/conftool/dbconfig/20260204-062055-marostegui.json |
[production] |
| 06:18 |
<marostegui@dns1006> |
END - running authdns-update |
[production] |
| 06:17 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2204 T416300', diff saved to https://phabricator.wikimedia.org/P88615 and previous config saved to /var/cache/conftool/dbconfig/20260204-061739-marostegui.json |
[production] |
| 06:17 |
<marostegui@dns1006> |
START - running authdns-update |
[production] |
| 06:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Promote db2207 to s2 primary and set section read-write T416300', diff saved to https://phabricator.wikimedia.org/P88614 and previous config saved to /var/cache/conftool/dbconfig/20260204-061637-marostegui.json |
[production] |
| 06:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set s2 codfw as read-only for maintenance - T416300', diff saved to https://phabricator.wikimedia.org/P88613 and previous config saved to /var/cache/conftool/dbconfig/20260204-061613-marostegui.json |
[production] |
| 06:12 |
<marostegui> |
Starting s2 codfw failover from db2204 to db2207 - T416300 |
[production] |
| 06:11 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s2 T416300 |
[production] |
| 06:11 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Set db2207 with weight 0 T416300', diff saved to https://phabricator.wikimedia.org/P88612 and previous config saved to /var/cache/conftool/dbconfig/20260204-061122-marostegui.json |
[production] |
| 06:10 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1235 (T415786)', diff saved to https://phabricator.wikimedia.org/P88611 and previous config saved to /var/cache/conftool/dbconfig/20260204-061047-marostegui.json |
[production] |
| 06:05 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2212 (T415786)', diff saved to https://phabricator.wikimedia.org/P88610 and previous config saved to /var/cache/conftool/dbconfig/20260204-060516-marostegui.json |
[production] |
| 06:05 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2212.codfw.wmnet with reason: Maintenance |
[production] |
| 05:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1231 (T415786)', diff saved to https://phabricator.wikimedia.org/P88609 and previous config saved to /var/cache/conftool/dbconfig/20260204-054542-marostegui.json |
[production] |
| 05:45 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1231.eqiad.wmnet with reason: Maintenance |
[production] |
| 05:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1227 (T415786)', diff saved to https://phabricator.wikimedia.org/P88608 and previous config saved to /var/cache/conftool/dbconfig/20260204-054518-marostegui.json |
[production] |
| 05:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P88607 and previous config saved to /var/cache/conftool/dbconfig/20260204-053009-marostegui.json |
[production] |
| 05:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P88606 and previous config saved to /var/cache/conftool/dbconfig/20260204-051501-marostegui.json |
[production] |
| 04:59 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1227 (T415786)', diff saved to https://phabricator.wikimedia.org/P88605 and previous config saved to /var/cache/conftool/dbconfig/20260204-045953-marostegui.json |
[production] |
| 04:41 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2202.codfw.wmnet with reason: Maintenance |
[production] |
| 04:41 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2188 (T415786)', diff saved to https://phabricator.wikimedia.org/P88604 and previous config saved to /var/cache/conftool/dbconfig/20260204-044137-marostegui.json |
[production] |
| 04:40 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1235 (T415786)', diff saved to https://phabricator.wikimedia.org/P88603 and previous config saved to /var/cache/conftool/dbconfig/20260204-044022-marostegui.json |
[production] |
| 04:40 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1235.eqiad.wmnet with reason: Maintenance |
[production] |
| 04:39 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1234 (T415786)', diff saved to https://phabricator.wikimedia.org/P88602 and previous config saved to /var/cache/conftool/dbconfig/20260204-043958-marostegui.json |
[production] |
| 04:29 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P88601 and previous config saved to /var/cache/conftool/dbconfig/20260204-042950-marostegui.json |
[production] |
| 04:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P88600 and previous config saved to /var/cache/conftool/dbconfig/20260204-042629-marostegui.json |
[production] |
| 04:19 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P88599 and previous config saved to /var/cache/conftool/dbconfig/20260204-041941-marostegui.json |
[production] |
| 04:11 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P88598 and previous config saved to /var/cache/conftool/dbconfig/20260204-041121-marostegui.json |
[production] |
| 04:09 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1234 (T415786)', diff saved to https://phabricator.wikimedia.org/P88597 and previous config saved to /var/cache/conftool/dbconfig/20260204-040933-marostegui.json |
[production] |
| 03:56 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2188 (T415786)', diff saved to https://phabricator.wikimedia.org/P88596 and previous config saved to /var/cache/conftool/dbconfig/20260204-035612-marostegui.json |
[production] |
| 03:31 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1227 (T415786)', diff saved to https://phabricator.wikimedia.org/P88595 and previous config saved to /var/cache/conftool/dbconfig/20260204-033110-marostegui.json |
[production] |
| 03:31 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1227.eqiad.wmnet with reason: Maintenance |
[production] |
| 03:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1202 (T415786)', diff saved to https://phabricator.wikimedia.org/P88594 and previous config saved to /var/cache/conftool/dbconfig/20260204-033046-marostegui.json |
[production] |
| 03:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P88593 and previous config saved to /var/cache/conftool/dbconfig/20260204-031537-marostegui.json |
[production] |
| 03:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P88592 and previous config saved to /var/cache/conftool/dbconfig/20260204-030029-marostegui.json |
[production] |
| 02:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1202 (T415786)', diff saved to https://phabricator.wikimedia.org/P88591 and previous config saved to /var/cache/conftool/dbconfig/20260204-024521-marostegui.json |
[production] |
| 02:36 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1234 (T415786)', diff saved to https://phabricator.wikimedia.org/P88590 and previous config saved to /var/cache/conftool/dbconfig/20260204-023659-marostegui.json |
[production] |
| 02:36 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1234.eqiad.wmnet with reason: Maintenance |
[production] |
| 02:36 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1232 (T415786)', diff saved to https://phabricator.wikimedia.org/P88589 and previous config saved to /var/cache/conftool/dbconfig/20260204-023634-marostegui.json |
[production] |
| 02:27 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2188 (T415786)', diff saved to https://phabricator.wikimedia.org/P88588 and previous config saved to /var/cache/conftool/dbconfig/20260204-022717-marostegui.json |
[production] |