|
2025-05-29
ยง
|
| 16:35 |
<bvibber@deploy1003> |
bvibber: Continuing with sync |
[production] |
| 16:33 |
<bvibber@deploy1003> |
bvibber: Backport for [[gerrit:1151787|Lua transform backend for JsonConfig Data: pages (T388434)]], [[gerrit:1151788|Chart-side support for Lua transforms (T388616)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 16:25 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 16:25 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns entries for sretest2007 - cmooney@cumin1002" |
[production] |
| 16:25 |
<cmooney@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: update dns entries for sretest2007 - cmooney@cumin1002" |
[production] |
| 16:25 |
<bking@cumin2002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on cirrussearch[2112-2113].codfw.wmnet with reason: firmware update |
[production] |
| 16:24 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cirrussearch[2111-2112].codfw.wmnet |
[production] |
| 16:24 |
<bking@cumin2002> |
START - Cookbook sre.hosts.remove-downtime for cirrussearch[2111-2112].codfw.wmnet |
[production] |
| 16:23 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P76682 and previous config saved to /var/cache/conftool/dbconfig/20250529-162305-fceratto.json |
[production] |
| 16:19 |
<cmooney@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:19 |
<cmooney@cumin1002> |
END (FAIL) - Cookbook sre.dns.netbox (exit_code=99) |
[production] |
| 16:17 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Banning hosts: cirrussearch2112*,cirrussearch2113* for T394543 - bking@cumin2002 |
[production] |
| 16:17 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2007.codfw.wmnet with OS bookworm |
[production] |
| 16:17 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cmooney@cumin1002" |
[production] |
| 16:17 |
<bking@cumin2002> |
START - Cookbook sre.elasticsearch.ban Banning hosts: cirrussearch2112*,cirrussearch2113* for T394543 - bking@cumin2002 |
[production] |
| 16:17 |
<cmooney@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - cmooney@cumin1002" |
[production] |
| 16:16 |
<cmooney@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 16:09 |
<bvibber@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1151787|Lua transform backend for JsonConfig Data: pages (T388434)]], [[gerrit:1151788|Chart-side support for Lua transforms (T388616)]] |
[production] |
| 16:07 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1175 (T395241)', diff saved to https://phabricator.wikimedia.org/P76681 and previous config saved to /var/cache/conftool/dbconfig/20250529-160758-fceratto.json |
[production] |
| 15:58 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1175 (T395241)', diff saved to https://phabricator.wikimedia.org/P76680 and previous config saved to /var/cache/conftool/dbconfig/20250529-155843-fceratto.json |
[production] |
| 15:58 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1175.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:58 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T395241)', diff saved to https://phabricator.wikimedia.org/P76679 and previous config saved to /var/cache/conftool/dbconfig/20250529-155817-fceratto.json |
[production] |
| 15:56 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2007.codfw.wmnet with reason: host reimage |
[production] |
| 15:56 |
<bvibber@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1151750|Enable Chart for Phase 4 wikis (all remaining public wikis) (T393788)]] (duration: 10m 20s) |
[production] |
| 15:52 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2007.codfw.wmnet with reason: host reimage |
[production] |
| 15:48 |
<bvibber@deploy1003> |
jforrester, bvibber: Continuing with sync |
[production] |
| 15:47 |
<bvibber@deploy1003> |
jforrester, bvibber: Backport for [[gerrit:1151750|Enable Chart for Phase 4 wikis (all remaining public wikis) (T393788)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 15:45 |
<bvibber@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1151750|Enable Chart for Phase 4 wikis (all remaining public wikis) (T393788)]] |
[production] |
| 15:43 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P76678 and previous config saved to /var/cache/conftool/dbconfig/20250529-154309-fceratto.json |
[production] |
| 15:34 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.reimage for host sretest2007.codfw.wmnet with OS bookworm |
[production] |
| 15:32 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 15:28 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P76677 and previous config saved to /var/cache/conftool/dbconfig/20250529-152802-fceratto.json |
[production] |
| 15:25 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 15:12 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T395241)', diff saved to https://phabricator.wikimedia.org/P76676 and previous config saved to /var/cache/conftool/dbconfig/20250529-151255-fceratto.json |
[production] |
| 15:03 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1166 (T395241)', diff saved to https://phabricator.wikimedia.org/P76675 and previous config saved to /var/cache/conftool/dbconfig/20250529-150332-fceratto.json |
[production] |
| 15:03 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:03 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157 (T395241)', diff saved to https://phabricator.wikimedia.org/P76674 and previous config saved to /var/cache/conftool/dbconfig/20250529-150306-fceratto.json |
[production] |
| 14:55 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:54 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:53 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:52 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:48 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:47 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P76673 and previous config saved to /var/cache/conftool/dbconfig/20250529-144759-fceratto.json |
[production] |
| 14:47 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:32 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P76672 and previous config saved to /var/cache/conftool/dbconfig/20250529-143251-fceratto.json |
[production] |
| 14:31 |
<dcausse> |
T395546: rebuildind eqiad completion indices for cswikiversity pnbwiktionary avkwiki mlwikiquote ladwiki ptwikiquote cswikiversity olowiki azwikibooks wikimania2018wiki rowikibooks rswikimedia liwiktionary oswiki nnwiktionary trwikisource vowikibooks ndswiktionary bmwikiquote ilowiki pawiktionary nowikibooks zhwikiversity tawikiquote hawiktionary akwikibooks udmwiki xhwikibooks |
[production] |
| 14:17 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157 (T395241)', diff saved to https://phabricator.wikimedia.org/P76671 and previous config saved to /var/cache/conftool/dbconfig/20250529-141744-fceratto.json |
[production] |
| 14:08 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1157 (T395241)', diff saved to https://phabricator.wikimedia.org/P76670 and previous config saved to /var/cache/conftool/dbconfig/20250529-140811-fceratto.json |
[production] |
| 14:08 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:39 |
<marostegui> |
Reboot dbproxy[1022,1025,1028-1029].eqiad.wmnet T395241 |
[production] |