1-50 of 10000 results (78ms)
2026-02-05 §
01:14 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P88678 and previous config saved to /var/cache/conftool/dbconfig/20260205-011410-marostegui.json [production]
01:06 <samwilson@deploy2002> Finished scap sync-world: Backport for [[gerrit:1236862|jquery.wikiEditor.js: disable resizing bar on proofread-page (T393231)]] (duration: 08m 21s) [production]
01:02 <samwilson@deploy2002> samwilson: Continuing with sync [production]
01:00 <samwilson@deploy2002> samwilson: Backport for [[gerrit:1236862|jquery.wikiEditor.js: disable resizing bar on proofread-page (T393231)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
00:59 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P88677 and previous config saved to /var/cache/conftool/dbconfig/20260205-005902-marostegui.json [production]
00:57 <samwilson@deploy2002> Started scap sync-world: Backport for [[gerrit:1236862|jquery.wikiEditor.js: disable resizing bar on proofread-page (T393231)]] [production]
00:51 <ryankemper@deploy2002> helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
00:51 <ryankemper@deploy2002> helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
00:43 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2208 (T415786)', diff saved to https://phabricator.wikimedia.org/P88676 and previous config saved to /var/cache/conftool/dbconfig/20260205-004353-marostegui.json [production]
00:36 <reedy@deploy2002> Finished scap sync-world: Backport for [[gerrit:1236692|Revert "Updated lcobucci/jwt from 4.1.5 to 4.3.0" (T416456)]], [[gerrit:1236693|Revert "Updated lcobucci/jwt from 4.1.5 to 4.3.0" (T416456)]] (duration: 06m 50s) [production]
00:32 <reedy@deploy2002> reedy, zabe: Continuing with sync [production]
00:32 <reedy@deploy2002> reedy, zabe: Backport for [[gerrit:1236692|Revert "Updated lcobucci/jwt from 4.1.5 to 4.3.0" (T416456)]], [[gerrit:1236693|Revert "Updated lcobucci/jwt from 4.1.5 to 4.3.0" (T416456)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
00:30 <reedy@deploy2002> Started scap sync-world: Backport for [[gerrit:1236692|Revert "Updated lcobucci/jwt from 4.1.5 to 4.3.0" (T416456)]], [[gerrit:1236693|Revert "Updated lcobucci/jwt from 4.1.5 to 4.3.0" (T416456)]] [production]
2026-02-04 §
23:16 <marostegui@cumin1003> dbctl commit (dc=all): 'Depooling db2208 (T415786)', diff saved to https://phabricator.wikimedia.org/P88674 and previous config saved to /var/cache/conftool/dbconfig/20260204-231600-marostegui.json [production]
23:15 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2208.codfw.wmnet with reason: Maintenance [production]
23:10 <eevans@cumin1003> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Applying upgrade to Java 11.0.30 — T416492 - eevans@cumin1003 [production]
22:28 <ryankemper@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
22:27 <ryankemper@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
22:27 <ryankemper@deploy2002> helmfile [dse-k8s-codfw] DONE helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
22:26 <ryankemper@deploy2002> helmfile [dse-k8s-codfw] START helmfile.d/dse-k8s-services/opensearch-semantic-search: apply [production]
22:03 <tgr_> late UTC deploys done [production]
22:02 <tgr@deploy2002> Finished scap sync-world: Backport for [[gerrit:1235551|Migrate EmailAuth config, step 1 (T404334)]] (duration: 11m 28s) [production]
21:56 <tgr@deploy2002> tgr: Continuing with sync [production]
21:55 <tgr@deploy2002> tgr: Backport for [[gerrit:1235551|Migrate EmailAuth config, step 1 (T404334)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
21:51 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2200.codfw.wmnet with reason: Maintenance [production]
21:51 <tgr@deploy2002> Started scap sync-world: Backport for [[gerrit:1235551|Migrate EmailAuth config, step 1 (T404334)]] [production]
21:47 <dancy@deploy2002> Finished scap sync-world: Backport for [[gerrit:1236811|Add messages for 'local-bot' global group (T415588)]], [[gerrit:1236812|Add messages for 'local-bot' global group (T415588)]] (duration: 40m 00s) [production]
21:34 <dancy@deploy2002> matmarex, dancy: Continuing with sync [production]
21:34 <dancy@deploy2002> matmarex, dancy: Backport for [[gerrit:1236811|Add messages for 'local-bot' global group (T415588)]], [[gerrit:1236812|Add messages for 'local-bot' global group (T415588)]] synced to the testservers (see https://wikitech.wikimedia.org/wiki/Mwdebug). Changes can now be verified there. [production]
21:07 <dancy@deploy2002> Started scap sync-world: Backport for [[gerrit:1236811|Add messages for 'local-bot' global group (T415588)]], [[gerrit:1236812|Add messages for 'local-bot' global group (T415588)]] [production]
21:06 <urandom> restart Cassandra to apply Java 11.0.30 upgrade, restbase/codfw — T416492 [production]
21:06 <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Applying upgrade to Java 11.0.30 — T416492 - eevans@cumin1003 [production]
20:52 <eevans@cumin1003> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Applying upgrade to Java 11.0.30 - eevans@cumin1003 [production]
20:05 <marostegui@cumin1003> DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2198.codfw.wmnet with reason: Maintenance [production]
20:05 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2182 (T415786)', diff saved to https://phabricator.wikimedia.org/P88673 and previous config saved to /var/cache/conftool/dbconfig/20260204-200512-marostegui.json [production]
19:52 <mvolz@deploy2002> helmfile [eqiad] DONE helmfile.d/services/zotero: apply [production]
19:52 <mvolz@deploy2002> helmfile [eqiad] START helmfile.d/services/zotero: apply [production]
19:51 <mvolz@deploy2002> helmfile [codfw] DONE helmfile.d/services/zotero: apply [production]
19:51 <mvolz@deploy2002> helmfile [codfw] START helmfile.d/services/zotero: apply [production]
19:50 <mvolz@deploy2002> helmfile [staging] DONE helmfile.d/services/zotero: apply [production]
19:50 <mvolz@deploy2002> helmfile [staging] START helmfile.d/services/zotero: apply [production]
19:50 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P88672 and previous config saved to /var/cache/conftool/dbconfig/20260204-195004-marostegui.json [production]
19:47 <cmooney@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
19:47 <cmooney@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns entries for fasw2-e16-eqiad pair - cmooney@cumin1003" [production]
19:47 <cmooney@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add dns entries for fasw2-e16-eqiad pair - cmooney@cumin1003" [production]
19:43 <cmooney@cumin1003> START - Cookbook sre.dns.netbox [production]
19:34 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P88671 and previous config saved to /var/cache/conftool/dbconfig/20260204-193455-marostegui.json [production]
19:19 <marostegui@cumin1003> dbctl commit (dc=all): 'Repooling after maintenance db2182 (T415786)', diff saved to https://phabricator.wikimedia.org/P88670 and previous config saved to /var/cache/conftool/dbconfig/20260204-191947-marostegui.json [production]
18:48 <urandom> restart Cassandra to apply Java 11.0.30 upgrade, restbase/eqiad — T416492 [production]
18:47 <eevans@cumin1003> START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Applying upgrade to Java 11.0.30 - eevans@cumin1003 [production]