|
2025-03-19
ยง
|
| 14:18 |
<root@deploy2002> |
helmfile [codfw] DONE helmfile.d/services/mw-jobrunner: sync |
[production] |
| 14:18 |
<root@deploy2002> |
helmfile [codfw] START helmfile.d/services/mw-jobrunner: sync |
[production] |
| 14:18 |
<hnowlan@cumin2002> |
START - Cookbook sre.switchdc.mediawiki.08-restart-mw-jobrunner for datacenter switchover from codfw to eqiad |
[production] |
| 14:17 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.07-set-readwrite (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:17 |
<hnowlan@cumin2002> |
MediaWiki read-only period ends at: 2025-03-19 14:17:55.451583 |
[production] |
| 14:17 |
<hnowlan@cumin2002> |
START - Cookbook sre.switchdc.mediawiki.07-set-readwrite for datacenter switchover from codfw to eqiad |
[production] |
| 14:17 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:17 |
<hnowlan@cumin2002> |
START - Cookbook sre.switchdc.mediawiki.06-set-db-readwrite for datacenter switchover from codfw to eqiad |
[production] |
| 14:17 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:16 |
<hnowlan@cumin2002> |
START - Cookbook sre.switchdc.mediawiki.04-switch-mediawiki for datacenter switchover from codfw to eqiad |
[production] |
| 14:16 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.03-set-db-readonly (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:15 |
<hnowlan@cumin2002> |
START - Cookbook sre.switchdc.mediawiki.03-set-db-readonly for datacenter switchover from codfw to eqiad |
[production] |
| 14:15 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.02-set-readonly (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:15 |
<hnowlan@cumin2002> |
MediaWiki read-only period starts at: 2025-03-19 14:15:30.955779 |
[production] |
| 14:15 |
<hnowlan@cumin2002> |
START - Cookbook sre.switchdc.mediawiki.02-set-readonly for datacenter switchover from codfw to eqiad |
[production] |
| 14:14 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:14 |
<hnowlan@cumin2002> |
START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from codfw to eqiad |
[production] |
| 14:13 |
<hnowlan@cumin2002> |
END (FAIL) - Cookbook sre.switchdc.mediawiki.01-stop-maintenance (exit_code=99) for datacenter switchover from codfw to eqiad |
[production] |
| 14:12 |
<hnowlan@cumin2002> |
START - Cookbook sre.switchdc.mediawiki.01-stop-maintenance for datacenter switchover from codfw to eqiad |
[production] |
| 14:12 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 14:12 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 14:11 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-reduce-ttl (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:07 |
<jclark@cumin1002> |
START - Cookbook sre.dns.netbox |
[production] |
| 14:06 |
<hnowlan@cumin2002> |
START - Cookbook sre.switchdc.mediawiki.00-reduce-ttl for datacenter switchover from codfw to eqiad |
[production] |
| 14:03 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:03 |
<hnowlan@cumin2002> |
START - Cookbook sre.switchdc.mediawiki.00-downtime-db-readonly-checks for datacenter switchover from codfw to eqiad |
[production] |
| 14:03 |
<hnowlan@cumin2002> |
END (PASS) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) for datacenter switchover from codfw to eqiad |
[production] |
| 14:03 |
<hnowlan@cumin2002> |
START - Cookbook sre.switchdc.mediawiki.00-disable-puppet for datacenter switchover from codfw to eqiad |
[production] |
| 13:59 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 13:59 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 13:58 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] DONE helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 13:58 |
<brouberol@deploy2002> |
helmfile [dse-k8s-eqiad] START helmfile.d/services/mediawiki-dumps-legacy: apply |
[production] |
| 13:55 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@deploy2002 /srv/mediawiki-staging (master $ u=) $ mwscript-k8s --follow --comment=T388158 -- cleanupTitles kaawiki |
[production] |
| 13:48 |
<hnowlan@deploy2002> |
Locking from deployment [ALL REPOSITORIES]: Datacenter Switchover - T385155 |
[production] |
| 13:45 |
<Lucas_WMDE> |
UTC afternoon backport+config window done |
[production] |
| 13:41 |
<Lucas_WMDE> |
lucaswerkmeister-wmde@deploy2002 /srv/mediawiki-staging (master $ u=) $ mwscript-k8s --comment=T388158 --follow -- namespaceDupes kaawiki --fix | tee ~/T388158 |
[production] |
| 13:40 |
<lucaswerkmeister-wmde@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1129204|Revert^4 "Add Portal namespace to kaawiki"]] (duration: 14m 01s) |
[production] |
| 13:33 |
<lucaswerkmeister-wmde@deploy2002> |
jhsoby, lucaswerkmeister-wmde: Continuing with sync |
[production] |
| 13:33 |
<lucaswerkmeister-wmde@deploy2002> |
jhsoby, lucaswerkmeister-wmde: Backport for [[gerrit:1129204|Revert^4 "Add Portal namespace to kaawiki"]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 13:26 |
<lucaswerkmeister-wmde@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1129204|Revert^4 "Add Portal namespace to kaawiki"]] |
[production] |
| 13:18 |
<jnuche@deploy2002> |
rebuilt and synchronized wikiversions files: group1 to 1.44.0-wmf.21 refs T386216 |
[production] |
| 13:12 |
<moritzm> |
installing sqlite3 security updates |
[production] |
| 12:47 |
<jnuche@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1129221|JobExecutor: Activate wrapping span (T389331)]] (duration: 15m 06s) |
[production] |
| 12:44 |
<XioNoX> |
trunk the sandbox vlan to ganeti500X - T385560 |
[production] |
| 12:40 |
<jnuche@deploy2002> |
kharlan, jnuche: Continuing with sync |
[production] |
| 12:39 |
<jnuche@deploy2002> |
kharlan, jnuche: Backport for [[gerrit:1129221|JobExecutor: Activate wrapping span (T389331)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 12:32 |
<jnuche@deploy2002> |
Started scap sync-world: Backport for [[gerrit:1129221|JobExecutor: Activate wrapping span (T389331)]] |
[production] |
| 12:26 |
<ladsgroup@deploy2002> |
Finished scap sync-world: Backport for [[gerrit:1129218|Bump thumbnail steps ratio to 25% (T360589)]] (duration: 16m 49s) |
[production] |
| 12:23 |
<moritzm> |
installing openjdk 17 security updates on puppet servers (the necessary restarts may cause a few interrupted puppet runs and will be splayed out) |
[production] |
| 12:19 |
<ladsgroup@deploy2002> |
ladsgroup: Continuing with sync |
[production] |