1251-1300 of 10000 results (29ms)
2025-05-06 §
09:28 <lucaswerkmeister-wmde@deploy1003> helmfile [codfw] START helmfile.d/services/wikidata-query-gui: apply [production]
09:28 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/airflow-analytics-test: apply [production]
09:27 <lucaswerkmeister-wmde@deploy1003> helmfile [eqiad] DONE helmfile.d/services/wikidata-query-gui: apply [production]
09:27 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/airflow-analytics-test: apply [production]
09:27 <lucaswerkmeister-wmde@deploy1003> helmfile [eqiad] START helmfile.d/services/wikidata-query-gui: apply [production]
09:26 <lucaswerkmeister-wmde@deploy1003> helmfile [staging] DONE helmfile.d/services/wikidata-query-gui: apply [production]
09:26 <lucaswerkmeister-wmde@deploy1003> helmfile [staging] START helmfile.d/services/wikidata-query-gui: apply [production]
07:49 <elukey> restart apache2 on puppetmaster1001 [production]
04:07 <mwpresync@deploy1003> Pruned MediaWiki: 1.44.0-wmf.24 (duration: 07m 35s) [production]
04:06 <mwpresync@deploy1003> Finished scap sync-world: testwikis to 1.44.0-wmf.28 refs T386223 (duration: 62m 44s) [production]
03:52 <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards [production]
03:45 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2015.codfw.wmnet, repooling source-only afterwards [production]
03:45 <ryankemper@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 15:00:00 on wdqs[2008,2014-2015].codfw.wmnet,wdqs[1011,1016].eqiad.wmnet with reason: T388134 [production]
03:44 <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards [production]
03:36 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2014.codfw.wmnet, repooling source-only afterwards [production]
03:35 <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling source-only afterwards [production]
03:27 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs2021.codfw.wmnet -> wdqs2008.codfw.wmnet, repooling source-only afterwards [production]
03:25 <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards [production]
03:18 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1011.eqiad.wmnet, repooling source-only afterwards [production]
03:18 <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@fe88851]: deploy to freshly reimaged host (duration: 00m 12s) [production]
03:17 <ryankemper@deploy1003> Started deploy [wdqs/wdqs@fe88851]: deploy to freshly reimaged host [production]
03:17 <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@fe88851]: deploy to freshly reimaged host (duration: 00m 13s) [production]
03:17 <ryankemper@deploy1003> Started deploy [wdqs/wdqs@fe88851]: deploy to freshly reimaged host [production]
03:17 <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@fe88851]: deploy to freshly reimaged host (duration: 00m 13s) [production]
03:16 <ryankemper@deploy1003> Started deploy [wdqs/wdqs@fe88851]: deploy to freshly reimaged host [production]
03:16 <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@fe88851]: deploy to freshly reimaged host (duration: 00m 14s) [production]
03:16 <ryankemper@deploy1003> Started deploy [wdqs/wdqs@fe88851]: deploy to freshly reimaged host [production]
03:13 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [admin]
03:05 <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
03:03 <mwpresync@deploy1003> Started scap sync-world: testwikis to 1.44.0-wmf.28 refs T386223 [production]
03:00 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
02:52 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T388134, bring new main graph hosts into service) xfer wikidata from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
02:49 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
02:43 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T388134, bring new main graph hosts into service) xfer wikidata from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
02:41 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wikidata from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
02:32 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T388134, bring new main graph hosts into service) xfer wdqs-all from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
02:27 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wdqs-all from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
02:24 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T388134, bring new main graph hosts into service) xfer wdqs-all from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
02:24 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wdqs-all from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
02:24 <ryankemper@deploy1003> Finished deploy [wdqs/wdqs@fe88851]: deploy to freshly reimaged host (duration: 00m 12s) [production]
02:24 <ryankemper@deploy1003> Started deploy [wdqs/wdqs@fe88851]: deploy to freshly reimaged host [production]
02:22 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T388134, bring new main graph hosts into service) xfer wdqs-all from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
02:22 <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T388134, bring new main graph hosts into service) xfer wdqs-all from wdqs1021.eqiad.wmnet -> wdqs1016.eqiad.wmnet, repooling source-only afterwards [production]
00:24 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host wdqs1017.eqiad.wmnet with OS bullseye [production]
2025-05-05 §
23:32 <ryankemper@cumin2002> START - Cookbook sre.hosts.reimage for host wdqs1017.eqiad.wmnet with OS bullseye [production]
23:29 <eileen> civicrm upgraded from 5a1f3e8e to 6ffbde61 [production]
23:14 <zabe> zabe@mwmaint1002:~$ mwscript extensions/WikimediaMaintenance/migrateESRefToContentTableStage2.php enwiki --delete /home/zabe/afl_text_table_deletedump/enwiki --sleep 0.3 # T381599 [production]
23:04 <zabe@deploy1003> Finished scap sync-world: Backport for [[gerrit:1140661|core-Permissions: refactor enwiki wgRemoveGroups]] (duration: 11m 13s) [production]
23:01 <ryankemper@cumin2002> END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['wdqs1017.eqiad.wmnet'] [production]
23:01 <ryankemper@cumin2002> START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['wdqs1017.eqiad.wmnet'] [production]