5301-5350 of 10000 results (30ms)
2024-09-09 §
08:20 <elukey@cumin2002> START - Cookbook sre.hosts.reimage for host dragonfly-supernode2001.codfw.wmnet with OS bookworm [production]
08:17 <arnaudb@cumin1002> dbctl commit (dc=all): 'Promote db2179 to s4 primary T373330', diff saved to https://phabricator.wikimedia.org/P68740 and previous config saved to /var/cache/conftool/dbconfig/20240909-081750-arnaudb.json [production]
08:17 <arnaudb> Starting s4 codfw failover from db2140 to db2179 - T373330 [production]
08:09 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
08:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'Remove db2179 from API/vslow/dump T373330', diff saved to https://phabricator.wikimedia.org/P68739 and previous config saved to /var/cache/conftool/dbconfig/20240909-080956-arnaudb.json [production]
08:09 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
08:09 <arnaudb@cumin1002> dbctl commit (dc=all): 'Set db2179 with weight 0 T373330', diff saved to https://phabricator.wikimedia.org/P68738 and previous config saved to /var/cache/conftool/dbconfig/20240909-080935-arnaudb.json [production]
08:09 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 32 hosts with reason: Primary switchover s4 T373330 [production]
08:08 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 32 hosts with reason: Primary switchover s4 T373330 [production]
08:05 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2220 T373175', diff saved to https://phabricator.wikimedia.org/P68737 and previous config saved to /var/cache/conftool/dbconfig/20240909-080558-arnaudb.json [production]
08:04 <jayme@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestage2002.codfw.wmnet with reason: host reimage [production]
08:04 <arnaudb@cumin1002> dbctl commit (dc=all): 'db2220 T373175', diff saved to https://phabricator.wikimedia.org/P68736 and previous config saved to /var/cache/conftool/dbconfig/20240909-080422-arnaudb.json [production]
08:01 <arnaudb@cumin1002> dbctl commit (dc=all): 'Promote db2218 to s7 primary T373175', diff saved to https://phabricator.wikimedia.org/P68735 and previous config saved to /var/cache/conftool/dbconfig/20240909-080108-arnaudb.json [production]
08:00 <arnaudb> Starting s7 codfw failover from db2220 to db2218 - T373175 [production]
08:00 <jayme@cumin1002> START - Cookbook sre.hosts.downtime for 2:00:00 on kubestage2002.codfw.wmnet with reason: host reimage [production]
07:58 <moritzm> installing openssl security updates [production]
07:52 <arnaudb@cumin1002> dbctl commit (dc=all): 'Remove db2218 from API/vslow/dump T373175', diff saved to https://phabricator.wikimedia.org/P68734 and previous config saved to /var/cache/conftool/dbconfig/20240909-075258-arnaudb.json [production]
07:51 <arnaudb@cumin1002> dbctl commit (dc=all): 'Set db2218 with weight 0 T373175', diff saved to https://phabricator.wikimedia.org/P68733 and previous config saved to /var/cache/conftool/dbconfig/20240909-075145-arnaudb.json [production]
07:51 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 29 hosts with reason: Primary switchover s7 T373175 [production]
07:51 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 1:00:00 on 29 hosts with reason: Primary switchover s7 T373175 [production]
07:39 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kubestage2002.codfw.wmnet with OS bookworm [production]
07:37 <jayme@cumin1002> END (FAIL) - Cookbook sre.k8s.renumber-node (exit_code=99) Renumbering for host kubestage2002.codfw.wmnet [production]
07:37 <jayme@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host kubestage2002.codfw.wmnet with OS bookworm [production]
07:37 <jayme@cumin1002> START - Cookbook sre.hosts.reimage for host kubestage2002.codfw.wmnet with OS bookworm [production]
07:36 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) depool for host kubestage2002.codfw.wmnet [production]
07:33 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-node depool for host kubestage2002.codfw.wmnet [production]
07:33 <jayme@cumin1002> START - Cookbook sre.k8s.renumber-node Renumbering for host kubestage2002.codfw.wmnet [production]
07:33 <jayme@cumin1002> END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host kubestage2001.codfw.wmnet [production]
07:33 <jayme@cumin1002> START - Cookbook sre.k8s.pool-depool-node pool for host kubestage2001.codfw.wmnet [production]
07:17 <moritzm> installing Linux 5.10.223 on bullseye hosts [production]
07:06 <moritzm> roll out debmonitor-client 0.4.0-2+deb11u1 on bullseye hosts [production]
06:56 <moritzm> installing aom security updates [production]
02:55 <wmbot~anticomposite@tools-bastion-13> Deploy d4122ba [tools.krinklebot]
2024-09-08 §
21:07 <wmbot~deltaquad@tools-bastion-12> ./stewardbots/StewardBot/manage.sh restart # Disconnected [tools.stewardbots]
21:06 <wmbot~deltaquad@tools-sgebastion-10> Restarted StewardBot/SULWatcher because of a connection loss [tools.stewardbots]
09:49 <wmbot~bsadowski1@tools-bastion-13> Restarted StewardBot/SULWatcher because of a connection loss [tools.stewardbots]
06:18 <wmbot~deltaquad@tools-bastion-12> ./stewardbots/StewardBot/manage.sh restart # Disconnected [tools.stewardbots]
2024-09-07 §
14:44 <ebernhardson@deploy1003> helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply [production]
14:44 <ebernhardson@deploy1003> helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply [production]
14:38 <dani@deploy1003> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
14:38 <dani@deploy1003> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
14:38 <dani@deploy1003> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
14:38 <dani@deploy1003> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
14:38 <dani@deploy1003> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
14:38 <dani@deploy1003> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
14:34 <dani@deploy1003> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
14:34 <dani@deploy1003> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
14:34 <dani@deploy1003> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
14:34 <dani@deploy1003> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
14:34 <dani@deploy1003> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]