|
2025-05-29
ยง
|
| 15:34 |
<cmooney@cumin1002> |
START - Cookbook sre.hosts.reimage for host sretest2007.codfw.wmnet with OS bookworm |
[production] |
| 15:32 |
<jhancock@cumin2002> |
END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 15:28 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P76677 and previous config saved to /var/cache/conftool/dbconfig/20250529-152802-fceratto.json |
[production] |
| 15:25 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 15:12 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1166 (T395241)', diff saved to https://phabricator.wikimedia.org/P76676 and previous config saved to /var/cache/conftool/dbconfig/20250529-151255-fceratto.json |
[production] |
| 15:03 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1166 (T395241)', diff saved to https://phabricator.wikimedia.org/P76675 and previous config saved to /var/cache/conftool/dbconfig/20250529-150332-fceratto.json |
[production] |
| 15:03 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1166.eqiad.wmnet with reason: Maintenance |
[production] |
| 15:03 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157 (T395241)', diff saved to https://phabricator.wikimedia.org/P76674 and previous config saved to /var/cache/conftool/dbconfig/20250529-150306-fceratto.json |
[production] |
| 14:55 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:54 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:53 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:52 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:48 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:47 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P76673 and previous config saved to /var/cache/conftool/dbconfig/20250529-144759-fceratto.json |
[production] |
| 14:47 |
<jhancock@cumin2002> |
START - Cookbook sre.hosts.provision for host sretest2007.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 14:32 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157', diff saved to https://phabricator.wikimedia.org/P76672 and previous config saved to /var/cache/conftool/dbconfig/20250529-143251-fceratto.json |
[production] |
| 14:31 |
<dcausse> |
T395546: rebuildind eqiad completion indices for cswikiversity pnbwiktionary avkwiki mlwikiquote ladwiki ptwikiquote cswikiversity olowiki azwikibooks wikimania2018wiki rowikibooks rswikimedia liwiktionary oswiki nnwiktionary trwikisource vowikibooks ndswiktionary bmwikiquote ilowiki pawiktionary nowikibooks zhwikiversity tawikiquote hawiktionary akwikibooks udmwiki xhwikibooks |
[production] |
| 14:17 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1157 (T395241)', diff saved to https://phabricator.wikimedia.org/P76671 and previous config saved to /var/cache/conftool/dbconfig/20250529-141744-fceratto.json |
[production] |
| 14:08 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1157 (T395241)', diff saved to https://phabricator.wikimedia.org/P76670 and previous config saved to /var/cache/conftool/dbconfig/20250529-140811-fceratto.json |
[production] |
| 14:08 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance |
[production] |
| 13:39 |
<marostegui> |
Reboot dbproxy[1022,1025,1028-1029].eqiad.wmnet T395241 |
[production] |
| 13:39 |
<marostegui@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbproxy[1022,1025,1028-1029].eqiad.wmnet with reason: Maintenance |
[production] |
| 13:39 |
<taavi@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1148299|Allow itwiki bureaucrat to remove sysop permission (T394752)]] (duration: 10m 13s) |
[production] |
| 13:32 |
<taavi@deploy1003> |
simmed, taavi: Continuing with sync |
[production] |
| 13:31 |
<taavi@deploy1003> |
simmed, taavi: Backport for [[gerrit:1148299|Allow itwiki bureaucrat to remove sysop permission (T394752)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 13:28 |
<taavi@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1148299|Allow itwiki bureaucrat to remove sysop permission (T394752)]] |
[production] |
| 13:26 |
<taavi@deploy1003> |
Finished scap sync-world: Backport for [[gerrit:1151813|Fixes issues with recommendations config in production (T393943)]] (duration: 12m 09s) |
[production] |
| 13:19 |
<taavi@deploy1003> |
jdlrobson, taavi: Continuing with sync |
[production] |
| 13:16 |
<taavi@deploy1003> |
jdlrobson, taavi: Backport for [[gerrit:1151813|Fixes issues with recommendations config in production (T393943)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. |
[production] |
| 13:14 |
<taavi@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1151813|Fixes issues with recommendations config in production (T393943)]] |
[production] |
| 12:56 |
<jclark@cumin1002> |
START - Cookbook sre.hosts.provision for host an-worker1186.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED |
[production] |
| 12:52 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 12:52 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 12:51 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 12:51 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 12:51 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 12:51 |
<btullis@deploy1003> |
helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 12:45 |
<dcausse> |
T395546: restoring pnbwiktionary avkwiki ladwiki ptwikiquote cswikiversity azwikibooks wikimania2018wiki liwiktionary nnwiktionary vowikibooks bmwikiquote nowikibooks zhwikiversity akwikibooks general indices in omega@eqiad from omega@codfw |
[production] |
| 12:38 |
<dcausse> |
T395546 (errata from previous log: s/psi/omega/) restoring collabwiki iewikibooks cywikiquote fawikibooks svwikiquote biwiktionary afwiktionary kywiktionary trwikisource mnwiktionary wikimania2007wiki sgwiki content indices in omega@eqiad from omega@codfw |
[production] |
| 12:38 |
<marostegui> |
Deploy schema change on s3 eqiad dbmaint with replication T395335 |
[production] |
| 12:37 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 12:37 |
<cmooney@cumin1002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns entries for cagefive2001 test server - cmooney@cumin1002" |
[production] |
| 12:37 |
<cmooney@cumin1002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns entries for cagefive2001 test server - cmooney@cumin1002" |
[production] |
| 12:36 |
<elukey@deploy1003> |
helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'sync'. |
[production] |
| 12:36 |
<elukey@deploy1003> |
helmfile [ml-serve-eqiad] START helmfile.d/admin 'sync'. |
[production] |
| 12:36 |
<marostegui> |
Deploy schema change on s7 eqiad dbmaint with replication T395335 |
[production] |
| 12:36 |
<dcausse> |
T395546 restoring collabwiki iewikibooks cywikiquote fawikibooks svwikiquote biwiktionary afwiktionary kywiktionary trwikisource mnwiktionary wikimania2007wiki sgwiki content indices in psi@eqiad from psi@codfw |
[production] |
| 12:34 |
<dcausse> |
T395546 restoring xhwikibooks_general in psi@eqiad from psi@codfw |
[production] |
| 12:33 |
<dcausse> |
T395546 restoring bowikibooks_content in psi@eqiad from psi@codfw |
[production] |
| 12:32 |
<elukey@deploy1003> |
helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . |
[production] |