7851-7900 of 10000 results (43ms)
2024-10-22 ยง
12:05 <Dreamy_Jazz> Running MediaModeration scan on all group0 wikis [production]
12:03 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P70478 and previous config saved to /var/cache/conftool/dbconfig/20241022-120356-ladsgroup.json [production]
12:03 <lucaswerkmeister-wmde@deploy2002> Finished scap sync-world: Backport for [[gerrit:1082180|tests: Don't depend on Message implementation details (T377778)]], [[gerrit:1082181|Update for Message/MessageValue changes (T377778)]] (duration: 15m 27s) [production]
12:02 <klausman@cumin2002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Java 11 security updates - klausman@cumin2002 [production]
11:57 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host wikikube-worker[2085-2086,2088-2089].codfw.wmnet [production]
11:57 <klausman@cumin2002> END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-codfw: Java 11 security updates - klausman@cumin2002 [production]
11:56 <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde: Continuing with sync [production]
11:55 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host wikikube-worker[2085-2086,2088-2089].codfw.wmnet [production]
11:55 <lucaswerkmeister-wmde@deploy2002> lucaswerkmeister-wmde: Backport for [[gerrit:1082180|tests: Don't depend on Message implementation details (T377778)]], [[gerrit:1082181|Update for Message/MessageValue changes (T377778)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
11:54 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2009.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
11:48 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
11:48 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P70477 and previous config saved to /var/cache/conftool/dbconfig/20241022-114849-ladsgroup.json [production]
11:48 <kart_> Updated cxserver to 2024-10-22-112806-production (T357950) [production]
11:47 <lucaswerkmeister-wmde@deploy2002> Started scap sync-world: Backport for [[gerrit:1082180|tests: Don't depend on Message implementation details (T377778)]], [[gerrit:1082181|Update for Message/MessageValue changes (T377778)]] [production]
11:47 <kartik@deploy2002> helmfile [eqiad] DONE helmfile.d/services/cxserver: apply [production]
11:46 <kartik@deploy2002> helmfile [eqiad] START helmfile.d/services/cxserver: apply [production]
11:46 <kartik@deploy2002> helmfile [codfw] DONE helmfile.d/services/cxserver: apply [production]
11:45 <kartik@deploy2002> helmfile [codfw] START helmfile.d/services/cxserver: apply [production]
11:44 <kartik@deploy2002> helmfile [staging] DONE helmfile.d/services/cxserver: apply [production]
11:44 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti2009.codfw.wmnet [production]
11:43 <kartik@deploy2002> helmfile [staging] START helmfile.d/services/cxserver: apply [production]
11:43 <jayme@cumin1002> END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) check for host wikikube-worker2085.codfw.wmnet [production]
11:43 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-node check for host wikikube-worker2085.codfw.wmnet [production]
11:42 <akosiaris@cloudcumin1001> END (PASS) - Cookbook wmcs.vps.remove_user_from_project (exit_code=0) for user 'faidon' [testlabs]
11:42 <akosiaris@cloudcumin1001> START - Cookbook wmcs.vps.remove_user_from_project for user 'faidon' [testlabs]
11:42 <akosiaris@cloudcumin1001> END (PASS) - Cookbook wmcs.vps.remove_user_from_project (exit_code=0) for user 'faidon' [swift]
11:41 <akosiaris@cloudcumin1001> START - Cookbook wmcs.vps.remove_user_from_project for user 'faidon' [swift]
11:41 <akosiaris> remove faidon from WMCS projects maps, visualeditor, swift, testlabs per his request. Keep the bastion project. cc paravoid [production]
11:41 <akosiaris@cloudcumin1001> END (PASS) - Cookbook wmcs.vps.remove_user_from_project (exit_code=0) for user 'faidon' [visualeditor]
11:40 <akosiaris@cloudcumin1001> START - Cookbook wmcs.vps.remove_user_from_project for user 'faidon' [visualeditor]
11:39 <akosiaris@cloudcumin1001> END (PASS) - Cookbook wmcs.vps.remove_user_from_project (exit_code=0) for user 'faidon' [maps]
11:39 <akosiaris@cloudcumin1001> START - Cookbook wmcs.vps.remove_user_from_project for user 'faidon' [maps]
11:39 <klausman@cumin2002> START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Java 11 security updates - klausman@cumin2002 [production]
11:34 <jayme@cumin1002> END (FAIL) - Cookbook sre.k8s.pool-depool-node (exit_code=99) check for host kubestagemaster2005.codfw.wmnet [production]
11:34 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-node check for host kubestagemaster2005.codfw.wmnet [production]
11:33 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1186 (T376905)', diff saved to https://phabricator.wikimedia.org/P70476 and previous config saved to /var/cache/conftool/dbconfig/20241022-113342-ladsgroup.json [production]
11:27 <moritzm> installing Java 11 security updates [production]
11:24 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1186 (T376905)', diff saved to https://phabricator.wikimedia.org/P70475 and previous config saved to /var/cache/conftool/dbconfig/20241022-112408-ladsgroup.json [production]
11:24 <ladsgroup@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance [production]
11:23 <ladsgroup@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance [production]
11:23 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T376905)', diff saved to https://phabricator.wikimedia.org/P70474 and previous config saved to /var/cache/conftool/dbconfig/20241022-112343-ladsgroup.json [production]
11:21 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/shellbox-video: sync [production]
11:21 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/shellbox-video: sync [production]
11:08 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P70473 and previous config saved to /var/cache/conftool/dbconfig/20241022-110836-ladsgroup.json [production]
11:07 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2149 (re)pooling @ 100%: post clone', diff saved to https://phabricator.wikimedia.org/P70472 and previous config saved to /var/cache/conftool/dbconfig/20241022-110744-arnaudb.json [production]
10:53 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1184', diff saved to https://phabricator.wikimedia.org/P70471 and previous config saved to /var/cache/conftool/dbconfig/20241022-105329-ladsgroup.json [production]
10:52 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2149 (re)pooling @ 75%: post clone', diff saved to https://phabricator.wikimedia.org/P70470 and previous config saved to /var/cache/conftool/dbconfig/20241022-105238-arnaudb.json [production]
10:38 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Repooling after maintenance db1184 (T376905)', diff saved to https://phabricator.wikimedia.org/P70469 and previous config saved to /var/cache/conftool/dbconfig/20241022-103822-ladsgroup.json [production]
10:37 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2149 (re)pooling @ 50%: post clone', diff saved to https://phabricator.wikimedia.org/P70468 and previous config saved to /var/cache/conftool/dbconfig/20241022-103733-arnaudb.json [production]
10:29 <ladsgroup@cumin1002> dbctl commit (dc=all): 'Depooling db1184 (T376905)', diff saved to https://phabricator.wikimedia.org/P70467 and previous config saved to /var/cache/conftool/dbconfig/20241022-102907-ladsgroup.json [production]