801-850 of 10000 results (147ms)
2026-05-26 ยง
17:24 <jclark@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding dse-k8s-wdqs1001 to eqiad - jclark@cumin1003" [production]
17:23 <fceratto@cumin1003> dbctl commit (dc=all): 'Depooling db2227 (T426633)', diff saved to https://phabricator.wikimedia.org/P93058 and previous config saved to /var/cache/conftool/dbconfig/20260526-172332-fceratto.json [production]
17:23 <fceratto@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2227.codfw.wmnet with reason: Maintenance [production]
17:23 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2209 (T426633)', diff saved to https://phabricator.wikimedia.org/P93057 and previous config saved to /var/cache/conftool/dbconfig/20260526-172303-fceratto.json [production]
17:21 <brett@cumin2002> cookbooks.sre.cdn.roll-reboot finished rebooting cp2045.codfw.wmnet [production]
17:20 <jclark@cumin1003> START - Cookbook sre.dns.netbox [production]
17:20 <brett@cumin2002> cookbooks.sre.cdn.roll-reboot finished rebooting cp2046.codfw.wmnet [production]
17:18 <jclark@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
17:17 <arlolra@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
17:17 <arlolra@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
17:17 <arlolra@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
17:17 <arlolra@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
17:17 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
17:17 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
17:17 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
17:16 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
17:16 <arlolra@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
17:16 <arlolra@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
17:16 <arlolra@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
17:16 <arlolra@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
17:15 <bking@cumin2002> START - Cookbook sre.elasticsearch.rolling-operation Operation.REBOOT (3 nodes at a time) for ElasticSearch cluster search_eqiad: T426585 - bking@cumin2002 [production]
17:14 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
17:14 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
17:14 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
17:14 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
17:13 <jclark@cumin1003> START - Cookbook sre.hosts.provision for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
17:13 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
17:13 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
17:13 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
17:13 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
17:12 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93056 and previous config saved to /var/cache/conftool/dbconfig/20260526-171255-fceratto.json [production]
17:11 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
17:11 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
17:11 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
17:11 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
17:09 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
17:09 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
17:09 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
17:09 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
17:07 <jclark@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host wdqs1038.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
17:05 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
17:05 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
17:05 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
17:05 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
17:02 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]
17:02 <fceratto@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P93055 and previous config saved to /var/cache/conftool/dbconfig/20260526-170247-fceratto.json [production]
17:02 <jgiannelos@deploy1003> helmfile [codfw] START helmfile.d/services/mw-parsoid: apply [production]
17:02 <jgiannelos@deploy1003> helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply [production]
17:02 <jgiannelos@deploy1003> helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply [production]
17:00 <jgiannelos@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-parsoid: apply [production]