1901-1950 of 10000 results (150ms)
2025-03-12 ยง
16:00 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.loadbalancer.admin (exit_code=0) config_reloading P{lvs6002.drmrs.wmnet} and A:liberica (T384477) [production]
16:00 <vgutierrez@cumin1002> START - Cookbook sre.loadbalancer.admin config_reloading P{lvs6002.drmrs.wmnet} and A:liberica (T384477) [production]
15:55 <jmm@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ganeti1034.eqiad.wmnet [production]
15:48 <jmm@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on ganeti1034.eqiad.wmnet with reason: remove from cluster for reimage [production]
15:44 <ladsgroup@deploy2002> Finished scap sync-world: Backport for [[gerrit:1126978|Bump the thumbnail steps ratio to 5% (T360589)]] (duration: 11m 30s) [production]
15:38 <ladsgroup@deploy2002> ladsgroup: Continuing with sync [production]
15:36 <ladsgroup@deploy2002> ladsgroup: Backport for [[gerrit:1126978|Bump the thumbnail steps ratio to 5% (T360589)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
15:33 <ladsgroup@deploy2002> Started scap sync-world: Backport for [[gerrit:1126978|Bump the thumbnail steps ratio to 5% (T360589)]] [production]
15:30 <mszabo@deploy2002> Finished scap sync-world: Backport for [[gerrit:1126979|GlobalUserSelectQueryBuilder: Ignore unattached local users (T388125)]], [[gerrit:1126982|http: Promote MultiHttpClient warnings to errors (T384717)]] (duration: 12m 01s) [production]
15:24 <mszabo@deploy2002> mszabo: Continuing with sync [production]
15:22 <mszabo@deploy2002> mszabo: Backport for [[gerrit:1126979|GlobalUserSelectQueryBuilder: Ignore unattached local users (T388125)]], [[gerrit:1126982|http: Promote MultiHttpClient warnings to errors (T384717)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
15:20 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti1034.eqiad.wmnet [production]
15:18 <mszabo@deploy2002> Started scap sync-world: Backport for [[gerrit:1126979|GlobalUserSelectQueryBuilder: Ignore unattached local users (T388125)]], [[gerrit:1126982|http: Promote MultiHttpClient warnings to errors (T384717)]] [production]
15:17 <Emperor> storcli64 /c0 restart on ms-be1090 T384003 [production]
15:14 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lvs6002.drmrs.wmnet with OS bookworm [production]
15:12 <elukey@deploy2002> helmfile [codfw] DONE helmfile.d/services/kartotherian: sync [production]
15:11 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for role: wdqs::internal_main@eqiad [production]
15:11 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [production]
15:10 <vgutierrez@cumin1002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-eqiad or A:lvs-secondary-eqiad) and A:bullseye and A:lvs [production]
15:10 <elukey@deploy2002> helmfile [codfw] START helmfile.d/services/kartotherian: sync [production]
15:06 <vgutierrez@cumin1002> START - Cookbook sre.loadbalancer.migrate-service-ipip for role: wdqs::internal_main@eqiad [production]
15:00 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.loadbalancer.migrate-service-ipip (exit_code=0) for role: wdqs::internal_main@codfw [production]
15:00 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.loadbalancer.restart-pybal (exit_code=0) rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [production]
14:59 <vgutierrez@cumin1002> START - Cookbook sre.loadbalancer.restart-pybal rolling-restart of pybal on (A:lvs-low-traffic-codfw or A:lvs-secondary-codfw) and A:bullseye and A:lvs [production]
14:55 <stevemunene@cumin1002> START - Cookbook sre.hadoop.init-hadoop-workers for hosts an-worker[1200-1208].eqiad.wmnet [production]
14:55 <stevemunene@cumin1002> END (PASS) - Cookbook sre.hadoop.init-hadoop-workers (exit_code=0) for hosts an-worker[1187-1199].eqiad.wmnet [production]
14:55 <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1126949|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126950|Replace distinct-values SPARQL queries (T369079)]], [[gerrit:1126951|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126952|Replace distinct-values SPARQL queries (T369079)]] (duration: 12m 58s) [production]
14:53 <vgutierrez@cumin1002> START - Cookbook sre.loadbalancer.migrate-service-ipip for role: wdqs::internal_main@codfw [production]
14:53 <vgutierrez@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lvs6002.drmrs.wmnet with reason: host reimage [production]
14:50 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply [production]
14:50 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply [production]
14:49 <vgutierrez@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on lvs6002.drmrs.wmnet with reason: host reimage [production]
14:48 <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde: Continuing with sync [production]
14:45 <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde: Backport for [[gerrit:1126949|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126950|Replace distinct-values SPARQL queries (T369079)]], [[gerrit:1126951|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126952|Replace distinct-values SPARQL queries (T369079)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:42 <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1126949|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126950|Replace distinct-values SPARQL queries (T369079)]], [[gerrit:1126951|Improve SPARQL query construction in SparqlHelper]], [[gerrit:1126952|Replace distinct-values SPARQL queries (T369079)]] [production]
14:40 <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1126577|Remove Flow as the default talk system (T383569)]] (duration: 11m 32s) [production]
14:37 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
14:36 <brouberol@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
14:33 <tgr@deploy2002> zoe, tgr: Continuing with sync [production]
14:32 <vgutierrez@cumin1002> START - Cookbook sre.hosts.reimage for host lvs6002.drmrs.wmnet with OS bookworm [production]
14:31 <tgr@deploy2002> zoe, tgr: Backport for [[gerrit:1126577|Remove Flow as the default talk system (T383569)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
14:30 <elukey@deploy2002> helmfile [staging] DONE helmfile.d/services/kartotherian: sync [production]
14:29 <elukey@deploy2002> helmfile [staging] START helmfile.d/services/kartotherian: sync [production]
14:28 <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1126577|Remove Flow as the default talk system (T383569)]] [production]
14:26 <vgutierrez@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on lvs6002.drmrs.wmnet with reason: depooled before reimage [production]
14:26 <jforrester@deploy2002> helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply [production]
14:26 <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1123435|Add MP event stream for MassDelete workflows (T382147)]], [[gerrit:1126648|Enable SUL3 signup for 50% of group 2 users (T384218)]], [[gerrit:1126956|[enwiki] Throttle exemption for event (T388637)]] (duration: 11m 04s) [production]
14:26 <jforrester@deploy2002> helmfile [eqiad] START helmfile.d/services/wikifunctions: apply [production]
14:26 <jforrester@deploy2002> helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply [production]
14:26 <vgutierrez> depooling lvs6002 before getting reimaged - T384477 [production]