|
2026-02-04
ยง
|
| 11:47 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2150 (T415786)', diff saved to https://phabricator.wikimedia.org/P88644 and previous config saved to /var/cache/conftool/dbconfig/20260204-114718-marostegui.json |
[production] |
| 11:45 |
<jynus@cumin1003> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1015.eqiad.wmnet with reason: host reimage |
[production] |
| 11:42 |
<moritzm> |
installing openjdk-11 security updates |
[production] |
| 11:41 |
<jynus@cumin1003> |
START - Cookbook sre.hosts.downtime for 2:00:00 on backup1015.eqiad.wmnet with reason: host reimage |
[production] |
| 11:39 |
<elukey@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/thumbor: sync |
[production] |
| 11:39 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance |
[production] |
| 11:38 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1251 (T415786)', diff saved to https://phabricator.wikimedia.org/P88643 and previous config saved to /var/cache/conftool/dbconfig/20260204-113854-marostegui.json |
[production] |
| 11:38 |
<elukey@deploy2002> |
helmfile [codfw] START helmfile.d/services/thumbor: sync |
[production] |
| 11:35 |
<elukey@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/thumbor: sync |
[production] |
| 11:35 |
<elukey@deploy2002> |
helmfile [eqiad] START helmfile.d/services/thumbor: sync |
[production] |
| 11:32 |
<elukey@deploy2002> |
helmfile [staging] DONE helmfile.d/services/thumbor: sync |
[production] |
| 11:32 |
<elukey@deploy2002> |
helmfile [staging] START helmfile.d/services/thumbor: sync |
[production] |
| 11:28 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P88642 and previous config saved to /var/cache/conftool/dbconfig/20260204-112846-marostegui.json |
[production] |
| 11:26 |
<jynus@cumin1003> |
START - Cookbook sre.hosts.reimage for host backup1015.eqiad.wmnet with OS trixie |
[production] |
| 11:23 |
<elukey@cumin1003> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host backup1015.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 11:22 |
<elukey@cumin1003> |
START - Cookbook sre.hosts.provision for host backup1015.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 11:18 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1251', diff saved to https://phabricator.wikimedia.org/P88641 and previous config saved to /var/cache/conftool/dbconfig/20260204-111837-marostegui.json |
[production] |
| 11:08 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1251 (T415786)', diff saved to https://phabricator.wikimedia.org/P88640 and previous config saved to /var/cache/conftool/dbconfig/20260204-110829-marostegui.json |
[production] |
| 10:48 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudcumin2001.codfw.wmnet |
[production] |
| 10:47 |
<moritzm> |
installing openjdk-17 security updates |
[production] |
| 10:44 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudcumin2001.codfw.wmnet |
[production] |
| 10:40 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2216 (T415786)', diff saved to https://phabricator.wikimedia.org/P88639 and previous config saved to /var/cache/conftool/dbconfig/20260204-104035-marostegui.json |
[production] |
| 10:39 |
<moritzm> |
upgrade cloudcumin2001 to bookworm T403153 |
[production] |
| 10:36 |
<hashar@deploy2002> |
rebuilt and synchronized wikiversions files: group0 to 1.46.0-wmf.14 refs T413805 |
[production] |
| 10:29 |
<hashar> |
Rolling back to group0 due to an issue with OAuth on metawiki # T413805 |
[production] |
| 10:25 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P88638 and previous config saved to /var/cache/conftool/dbconfig/20260204-102527-marostegui.json |
[production] |
| 10:11 |
<hashar> |
Restarted CI Jenkins |
[production] |
| 10:10 |
<hashar> |
Gerrit is back |
[production] |
| 10:10 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P88637 and previous config saved to /var/cache/conftool/dbconfig/20260204-101018-marostegui.json |
[production] |
| 10:06 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2150 (T415786)', diff saved to https://phabricator.wikimedia.org/P88636 and previous config saved to /var/cache/conftool/dbconfig/20260204-100638-marostegui.json |
[production] |
| 10:06 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2150.codfw.wmnet with reason: Maintenance |
[production] |
| 10:06 |
<hashar> |
Restarting Gerrit instances |
[production] |
| 09:55 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2216 (T415786)', diff saved to https://phabricator.wikimedia.org/P88635 and previous config saved to /var/cache/conftool/dbconfig/20260204-095510-marostegui.json |
[production] |
| 09:38 |
<moritzm> |
installing openssl security updates |
[production] |
| 09:37 |
<hashar@deploy2002> |
rebuilt and synchronized wikiversions files: group1 to 1.46.0-wmf.14 refs T413805 |
[production] |
| 09:34 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1251 (T415786)', diff saved to https://phabricator.wikimedia.org/P88634 and previous config saved to /var/cache/conftool/dbconfig/20260204-093421-marostegui.json |
[production] |
| 09:34 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1251.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:10 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance |
[production] |
| 09:10 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253 (T415786)', diff saved to https://phabricator.wikimedia.org/P88632 and previous config saved to /var/cache/conftool/dbconfig/20260204-091015-marostegui.json |
[production] |
| 08:55 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P88631 and previous config saved to /var/cache/conftool/dbconfig/20260204-085506-marostegui.json |
[production] |
| 08:39 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253', diff saved to https://phabricator.wikimedia.org/P88630 and previous config saved to /var/cache/conftool/dbconfig/20260204-083958-marostegui.json |
[production] |
| 08:24 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1253 (T415786)', diff saved to https://phabricator.wikimedia.org/P88629 and previous config saved to /var/cache/conftool/dbconfig/20260204-082450-marostegui.json |
[production] |
| 08:23 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db2216 (T415786)', diff saved to https://phabricator.wikimedia.org/P88628 and previous config saved to /var/cache/conftool/dbconfig/20260204-082324-marostegui.json |
[production] |
| 08:23 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2216.codfw.wmnet with reason: Maintenance |
[production] |
| 08:23 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2212 (T415786)', diff saved to https://phabricator.wikimedia.org/P88627 and previous config saved to /var/cache/conftool/dbconfig/20260204-082259-marostegui.json |
[production] |
| 08:08 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:07 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P88626 and previous config saved to /var/cache/conftool/dbconfig/20260204-080751-marostegui.json |
[production] |
| 07:52 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P88625 and previous config saved to /var/cache/conftool/dbconfig/20260204-075243-marostegui.json |
[production] |
| 07:39 |
<ryankemper@cumin2002> |
END (FAIL) - Cookbook sre.hadoop.reboot-workers (exit_code=99) for Hadoop analytics cluster |
[production] |
| 07:37 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db2212 (T415786)', diff saved to https://phabricator.wikimedia.org/P88624 and previous config saved to /var/cache/conftool/dbconfig/20260204-073735-marostegui.json |
[production] |