|
2026-03-25
§
|
| 15:07 |
<blake@cumin1003> |
END (PASS) - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 15:07 |
<root@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: sync |
[production] |
| 15:07 |
<root@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-jobrunner: sync |
[production] |
| 15:07 |
<blake@cumin1003> |
START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from codfw to eqiad |
[production] |
| 15:02 |
<blake@cumin1003> |
END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 15:02 |
<blake@cumin1003> |
MediaWiki read-only period ends at: 2026-03-25 15:02:52.921926 |
[production] |
| 15:02 |
<blake@cumin1003> |
START - Cookbook sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from codfw to eqiad |
[production] |
| 15:02 |
<blake@cumin1003> |
END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 15:02 |
<blake@cumin1003> |
START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from codfw to eqiad |
[production] |
| 15:02 |
<blake@cumin1003> |
END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 15:01 |
<blake@cumin1003> |
START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from codfw to eqiad |
[production] |
| 15:01 |
<blake@cumin1003> |
END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 15:00 |
<blake@cumin1003> |
START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from codfw to eqiad |
[production] |
| 15:00 |
<blake@cumin1003> |
END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 15:00 |
<blake@cumin1003> |
MediaWiki read-only period starts at: 2026-03-25 15:00:24.089398 |
[production] |
| 15:00 |
<blake@cumin1003> |
START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from codfw to eqiad |
[production] |
| 14:55 |
<blake@cumin1003> |
END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:53 |
<blake@cumin1003> |
START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from codfw to eqiad |
[production] |
| 14:52 |
<blake@cumin1003> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:52 |
<blake@cumin1003> |
START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from codfw to eqiad |
[production] |
| 14:51 |
<blake@cumin1003> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:46 |
<blake@cumin1003> |
START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from codfw to eqiad |
[production] |
| 14:28 |
<cdanis> |
💙cdanis@apt1002.wikimedia.org ~ 🕥☕ sudo -i reprepro --component main --restrict cidergrinder update trixie-wikimedia |
[production] |
| 14:28 |
<cdanis> |
💙cdanis@apt1002.wikimedia.org ~ 🕥☕ sudo -i reprepro --component main --restrict cidergrinder update bullseye-wikimedia |
[production] |
| 14:24 |
<jhancock@cumin2002> |
END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['phab2002'] |
[production] |
| 14:24 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. |
[production] |
| 14:23 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. |
[production] |
| 14:17 |
<jhancock@cumin2002> |
START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['phab2002'] |
[production] |
| 14:14 |
<gengh@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply |
[production] |
| 14:13 |
<gengh@deploy2002> |
helmfile [eqiad] START helmfile.d/services/wikifunctions: apply |
[production] |
| 14:13 |
<gengh@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply |
[production] |
| 14:12 |
<gengh@deploy2002> |
helmfile [codfw] START helmfile.d/services/wikifunctions: apply |
[production] |
| 14:11 |
<gengh@deploy2002> |
helmfile [staging] DONE helmfile.d/services/wikifunctions: apply |
[production] |
| 14:11 |
<dpogorzelski@deploy2002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:11 |
<dpogorzelski@deploy2002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:11 |
<gengh@deploy2002> |
helmfile [staging] START helmfile.d/services/wikifunctions: apply |
[production] |
| 14:10 |
<dpogorzelski@deploy2002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:10 |
<dpogorzelski@deploy2002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:08 |
<gengh@deploy2002> |
helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply |
[production] |
| 14:07 |
<gengh@deploy2002> |
helmfile [eqiad] START helmfile.d/services/wikifunctions: apply |
[production] |
| 14:07 |
<gengh@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply |
[production] |
| 14:07 |
<dpogorzelski@deploy2002> |
helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'. |
[production] |
| 14:06 |
<gengh@deploy2002> |
helmfile [codfw] START helmfile.d/services/wikifunctions: apply |
[production] |
| 14:06 |
<dpogorzelski@deploy2002> |
helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'. |
[production] |
| 14:06 |
<gengh@deploy2002> |
helmfile [staging] DONE helmfile.d/services/wikifunctions: apply |
[production] |
| 14:05 |
<gengh@deploy2002> |
helmfile [staging] START helmfile.d/services/wikifunctions: apply |
[production] |
| 14:00 |
<blake@cumin1003> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-lock-scap (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:00 |
<root@deploy2002> |
Locking from deployment [ALL REPOSITORIES]: Datacenter switchover from codfw to eqiad - |
[production] |
| 14:00 |
<blake@cumin1003> |
START - Cookbook sre.switchdc.mediawiki.00-lock-scap for datacenter switchover from codfw to eqiad |
[production] |
| 13:49 |
<otto@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1260091|EventStreamConfig - Increase spark_job_ingestion_scale for larger event streams (T360794 T351225)]] (duration: 07m 48s) |
[production] |