|
2025-11-05
ยง
|
| 08:29 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1181.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:29 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1174 (re)pooling @ 25%: After schema change', diff saved to https://phabricator.wikimedia.org/P84852 and previous config saved to /var/cache/conftool/dbconfig/20251105-082920-root.json |
[production] |
| 08:26 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1174 (T407997)', diff saved to https://phabricator.wikimedia.org/P84851 and previous config saved to /var/cache/conftool/dbconfig/20251105-082642-marostegui.json |
[production] |
| 08:25 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1174 (T407997)', diff saved to https://phabricator.wikimedia.org/P84850 and previous config saved to /var/cache/conftool/dbconfig/20251105-082533-marostegui.json |
[production] |
| 08:25 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1174.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:22 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 5%: Testing Debian Trixie in es2', diff saved to https://phabricator.wikimedia.org/P84849 and previous config saved to /var/cache/conftool/dbconfig/20251105-082209-root.json |
[production] |
| 08:21 |
<Emperor> |
run gitlab-package-puller by hand on apt-staging2001 |
[production] |
| 08:13 |
<brouberol@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on an-launcher1002.eqiad.wmnet with reason: host is being decommissioned |
[production] |
| 08:08 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1171.eqiad.wmnet with reason: Maintenance |
[production] |
| 08:08 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1170 (T407997)', diff saved to https://phabricator.wikimedia.org/P84848 and previous config saved to /var/cache/conftool/dbconfig/20251105-080849-marostegui.json |
[production] |
| 08:07 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 4%: Testing Debian Trixie in es2', diff saved to https://phabricator.wikimedia.org/P84847 and previous config saved to /var/cache/conftool/dbconfig/20251105-080702-root.json |
[production] |
| 08:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2212 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P84846 and previous config saved to /var/cache/conftool/dbconfig/20251105-080027-root.json |
[production] |
| 07:59 |
<eileen> |
ivicrm upgraded from 1eeb1a46 to 8efb2be1 |
[production] |
| 07:53 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P84845 and previous config saved to /var/cache/conftool/dbconfig/20251105-075341-marostegui.json |
[production] |
| 07:51 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 3%: Testing Debian Trixie in es2', diff saved to https://phabricator.wikimedia.org/P84844 and previous config saved to /var/cache/conftool/dbconfig/20251105-075156-root.json |
[production] |
| 07:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2212 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P84843 and previous config saved to /var/cache/conftool/dbconfig/20251105-074521-root.json |
[production] |
| 07:38 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P84842 and previous config saved to /var/cache/conftool/dbconfig/20251105-073833-marostegui.json |
[production] |
| 07:36 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 2%: Testing Debian Trixie in es2', diff saved to https://phabricator.wikimedia.org/P84841 and previous config saved to /var/cache/conftool/dbconfig/20251105-073651-root.json |
[production] |
| 07:33 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1203 (re)pooling @ 100%: 10', diff saved to https://phabricator.wikimedia.org/P84840 and previous config saved to /var/cache/conftool/dbconfig/20251105-073347-root.json |
[production] |
| 07:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2215 (re)pooling @ 100%: After switchover', diff saved to https://phabricator.wikimedia.org/P84839 and previous config saved to /var/cache/conftool/dbconfig/20251105-073033-root.json |
[production] |
| 07:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2212 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P84838 and previous config saved to /var/cache/conftool/dbconfig/20251105-073016-root.json |
[production] |
| 07:23 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1170 (T407997)', diff saved to https://phabricator.wikimedia.org/P84837 and previous config saved to /var/cache/conftool/dbconfig/20251105-072326-marostegui.json |
[production] |
| 07:21 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'es1033 (re)pooling @ 1%: Testing Debian Trixie in es2', diff saved to https://phabricator.wikimedia.org/P84836 and previous config saved to /var/cache/conftool/dbconfig/20251105-072145-root.json |
[production] |
| 07:18 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1203 (re)pooling @ 75%: 10', diff saved to https://phabricator.wikimedia.org/P84835 and previous config saved to /var/cache/conftool/dbconfig/20251105-071841-root.json |
[production] |
| 07:16 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Add es1033 to es2 depooled T409257 T407472', diff saved to https://phabricator.wikimedia.org/P84834 and previous config saved to /var/cache/conftool/dbconfig/20251105-071605-marostegui.json |
[production] |
| 07:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2215 (re)pooling @ 75%: After switchover', diff saved to https://phabricator.wikimedia.org/P84833 and previous config saved to /var/cache/conftool/dbconfig/20251105-071527-root.json |
[production] |
| 07:15 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2212 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P84832 and previous config saved to /var/cache/conftool/dbconfig/20251105-071510-root.json |
[production] |
| 07:07 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db2212 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P84831 and previous config saved to /var/cache/conftool/dbconfig/20251105-070707-marostegui.json |
[production] |
| 07:07 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2212.codfw.wmnet with reason: Maintenance |
[production] |
| 07:05 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depooling db1170 (T407997)', diff saved to https://phabricator.wikimedia.org/P84830 and previous config saved to /var/cache/conftool/dbconfig/20251105-070540-marostegui.json |
[production] |
| 07:05 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1170.eqiad.wmnet with reason: Maintenance |
[production] |
| 07:05 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1158 (T407997)', diff saved to https://phabricator.wikimedia.org/P84828 and previous config saved to /var/cache/conftool/dbconfig/20251105-070516-marostegui.json |
[production] |
| 07:03 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1203 (re)pooling @ 50%: 10', diff saved to https://phabricator.wikimedia.org/P84827 and previous config saved to /var/cache/conftool/dbconfig/20251105-070335-root.json |
[production] |
| 07:00 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2215 (re)pooling @ 50%: After switchover', diff saved to https://phabricator.wikimedia.org/P84826 and previous config saved to /var/cache/conftool/dbconfig/20251105-070021-root.json |
[production] |
| 06:52 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts es1034.eqiad.wmnet |
[production] |
| 06:52 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 06:52 |
<marostegui@cumin1003> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1034.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" |
[production] |
| 06:51 |
<marostegui@cumin1003> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: es1034.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1003" |
[production] |
| 06:50 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P84825 and previous config saved to /var/cache/conftool/dbconfig/20251105-065008-marostegui.json |
[production] |
| 06:48 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db1203 (re)pooling @ 25%: 10', diff saved to https://phabricator.wikimedia.org/P84824 and previous config saved to /var/cache/conftool/dbconfig/20251105-064829-root.json |
[production] |
| 06:48 |
<marostegui@cumin1003> |
START - Cookbook sre.dns.netbox |
[production] |
| 06:47 |
<eileen> |
civicrm upgraded from a7c697e9 to 1eeb1a46 |
[production] |
| 06:45 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2215 (re)pooling @ 25%: After switchover', diff saved to https://phabricator.wikimedia.org/P84823 and previous config saved to /var/cache/conftool/dbconfig/20251105-064515-root.json |
[production] |
| 06:42 |
<marostegui@cumin1003> |
START - Cookbook sre.hosts.decommission for hosts es1034.eqiad.wmnet |
[production] |
| 06:40 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Depool db1203 for migration to mariadb 10.11', diff saved to https://phabricator.wikimedia.org/P84822 and previous config saved to /var/cache/conftool/dbconfig/20251105-064028-marostegui.json |
[production] |
| 06:40 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1203.eqiad.wmnet with reason: Maintenance |
[production] |
| 06:35 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P84821 and previous config saved to /var/cache/conftool/dbconfig/20251105-063458-marostegui.json |
[production] |
| 06:32 |
<marostegui@cumin1003> |
DONE (ERROR) - Cookbook sre.hosts.downtime (exit_code=97) for 1:00:00 on 14 hosts with reason: Primary switchover x1 T409168 |
[production] |
| 06:32 |
<marostegui@cumin1003> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 14 hosts with reason: Primary switchover x1 T409168 |
[production] |
| 06:30 |
<marostegui@cumin1003> |
dbctl commit (dc=all): 'db2215 (re)pooling @ 10%: After switchover', diff saved to https://phabricator.wikimedia.org/P84820 and previous config saved to /var/cache/conftool/dbconfig/20251105-063009-root.json |
[production] |