1-50 of 10000 results (89ms)
2025-05-29 ยง
21:41 <cjming> end of UTC late backport window [production]
21:38 <cjming@deploy1003> Finished scap sync-world: Backport for [[gerrit:1152115|ext.wikimediaEvents: Add XLab PageVisit instrument (T393918 T392313)]] (duration: 09m 54s) [production]
21:31 <cjming@deploy1003> cjming: Continuing with sync [production]
21:30 <cjming@deploy1003> cjming: Backport for [[gerrit:1152115|ext.wikimediaEvents: Add XLab PageVisit instrument (T393918 T392313)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
21:28 <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1152115|ext.wikimediaEvents: Add XLab PageVisit instrument (T393918 T392313)]] [production]
21:20 <cjming@deploy1003> Finished scap sync-world: Backport for [[gerrit:1151148|EventStreamConfig: Remove xLab development streams (T393918)]] (duration: 09m 34s) [production]
21:12 <cjming@deploy1003> cjming, phuedx: Continuing with sync [production]
21:12 <cjming@deploy1003> cjming, phuedx: Backport for [[gerrit:1151148|EventStreamConfig: Remove xLab development streams (T393918)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
21:10 <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1151148|EventStreamConfig: Remove xLab development streams (T393918)]] [production]
21:07 <cjming@deploy1003> Finished scap sync-world: Backport for [[gerrit:1151236|noc: Fix invalid `max-age: 300` syntax to `max-age=300` in fileserve.php (T341859)]] (duration: 10m 22s) [production]
21:00 <cjming@deploy1003> cjming, krinkle: Continuing with sync [production]
20:59 <cjming@deploy1003> cjming, krinkle: Backport for [[gerrit:1151236|noc: Fix invalid `max-age: 300` syntax to `max-age=300` in fileserve.php (T341859)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
20:57 <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1151236|noc: Fix invalid `max-age: 300` syntax to `max-age=300` in fileserve.php (T341859)]] [production]
20:55 <cscott@deploy1003> Finished scap sync-world: Backport for [[gerrit:1152080|Campaign: Ensure `<div class="mw-parser-output">` wrapper is removed (T395023)]] (duration: 15m 01s) [production]
20:48 <cscott@deploy1003> cscott: Continuing with sync [production]
20:43 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance [production]
20:42 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T395241)', diff saved to https://phabricator.wikimedia.org/P76700 and previous config saved to /var/cache/conftool/dbconfig/20250529-204251-fceratto.json [production]
20:42 <cscott@deploy1003> cscott: Backport for [[gerrit:1152080|Campaign: Ensure `<div class="mw-parser-output">` wrapper is removed (T395023)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
20:40 <cscott@deploy1003> Started scap sync-world: Backport for [[gerrit:1152080|Campaign: Ensure `<div class="mw-parser-output">` wrapper is removed (T395023)]] [production]
20:28 <cjming@deploy1003> Finished scap sync-world: Backport for [[gerrit:1149720|Turn on glent m1 AB test (T262612)]] (duration: 10m 19s) [production]
20:27 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P76699 and previous config saved to /var/cache/conftool/dbconfig/20250529-202743-fceratto.json [production]
20:21 <cjming@deploy1003> ebernhardson, cjming: Continuing with sync [production]
20:20 <cjming@deploy1003> ebernhardson, cjming: Backport for [[gerrit:1149720|Turn on glent m1 AB test (T262612)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
20:18 <cjming@deploy1003> Started scap sync-world: Backport for [[gerrit:1149720|Turn on glent m1 AB test (T262612)]] [production]
20:16 <jdlrobson@deploy1003> Finished scap sync-world: Backport for [[gerrit:1151812|Enable Minerva typeahead search on beta cluster (T380510)]], [[gerrit:1151313|Enable ReadingList special page on test wiki]] (duration: 10m 13s) [production]
20:12 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P76698 and previous config saved to /var/cache/conftool/dbconfig/20250529-201236-fceratto.json [production]
20:08 <jdlrobson@deploy1003> jdlrobson: Continuing with sync [production]
20:07 <jdlrobson@deploy1003> jdlrobson: Backport for [[gerrit:1151812|Enable Minerva typeahead search on beta cluster (T380510)]], [[gerrit:1151313|Enable ReadingList special page on test wiki]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
20:05 <jdlrobson@deploy1003> Started scap sync-world: Backport for [[gerrit:1151812|Enable Minerva typeahead search on beta cluster (T380510)]], [[gerrit:1151313|Enable ReadingList special page on test wiki]] [production]
19:57 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1212 (T395241)', diff saved to https://phabricator.wikimedia.org/P76697 and previous config saved to /var/cache/conftool/dbconfig/20250529-195729-fceratto.json [production]
19:05 <bking@cumin2002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on relforge[1003-1004,1008-1009].eqiad.wmnet with reason: noisy alerts [production]
18:57 <fceratto@cumin1002> dbctl commit (dc=all): 'Depooling db1212 (T395241)', diff saved to https://phabricator.wikimedia.org/P76696 and previous config saved to /var/cache/conftool/dbconfig/20250529-185711-fceratto.json [production]
18:57 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1013,1017].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance [production]
18:56 <fceratto@cumin1002> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1212.eqiad.wmnet with reason: Maintenance [production]
18:56 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T395241)', diff saved to https://phabricator.wikimedia.org/P76695 and previous config saved to /var/cache/conftool/dbconfig/20250529-185639-fceratto.json [production]
18:41 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P76694 and previous config saved to /var/cache/conftool/dbconfig/20250529-184132-fceratto.json [production]
18:27 <dancy@deploy1003> rebuilt and synchronized wikiversions files: group2 to 1.45.0-wmf.3 refs T392173 [production]
18:27 <dr0ptp4kt@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic: apply [production]
18:26 <dr0ptp4kt@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic: apply [production]
18:26 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P76693 and previous config saved to /var/cache/conftool/dbconfig/20250529-182624-fceratto.json [production]
18:22 <dr0ptp4kt@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply [production]
18:21 <fceratto@deploy1003> helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [production]
18:21 <dr0ptp4kt@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply [production]
18:21 <fceratto@deploy1003> helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [production]
18:12 <fceratto@deploy1003> helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [production]
18:12 <fceratto@deploy1003> helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [production]
18:11 <fceratto@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1198 (T395241)', diff saved to https://phabricator.wikimedia.org/P76692 and previous config saved to /var/cache/conftool/dbconfig/20250529-181118-fceratto.json [production]
18:01 <fceratto@deploy1003> helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [production]
18:01 <fceratto@deploy1003> helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [production]
17:53 <fceratto@deploy1003> helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [production]