1-50 of 10000 results (24ms)
2025-06-29 §
13:50 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'. [production]
13:50 <brouberol@deploy1003> helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. [production]
03:30 <wmbot~anticomposite@tools-bastion-13> git remote set-url origin https://gitlab.wikimedia.org/toolforge-repos/bullseye.git [tools.bullseye]
03:28 <wmbot~anticomposite@tools-bastion-13> Deploy https://gitlab.wikimedia.org/toolforge-repos/bullseye/-/commit/eb4b84e71fccea7a73991c6e4d7a0b78bb318490 [tools.bullseye]
2025-06-28 §
20:49 <wmbot~soda@tools-bastion-13> soda built and uploaded a new version [tools.yapping-sodium]
20:14 <wmbot~soda@tools-bastion-13> soda built and uploaded a new version [tools.yapping-sodium]
19:47 <wmbot~soda@tools-bastion-13> soda built and uploaded a new version [tools.yapping-sodium]
10:39 <dcaro@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24 [tools]
10:13 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19, tools-k8s-worker-nfs-67, tools-k8s-worker-nfs-43, tools-k8s-worker-nfs-22, tools-k8s-worker-nfs-5, tools-k8s-worker-nfs-24 [tools]
10:13 <dcaro@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=99) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24 [tools]
10:13 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67,tools-k8s-worker-nfs-43,tools-k8s-worker-nfs-22,tools-k8s-worker-nfs-5,tools-k8s-worker-nfs-24 [tools]
10:12 <dcaro@cloudcumin1001> END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67 [tools]
10:12 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19,tools-k8s-worker-nfs-67 [tools]
10:12 <dcaro@cloudcumin1001> END (ERROR) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=97) for tools-k8s-worker-nfs-67 [tools]
10:12 <dcaro@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-67 [tools]
10:08 <dcaro> left a tmux running with a script to restart nginx if stuck [tools]
09:58 <dcaro> restarted nginx in tools-static [tools]
2025-06-27 §
23:21 <cwhite> truncate /var/log/syslog on prometheus1005 T398091 [production]
22:43 <urbanecm> Start `GrowthExperiments:revalidateLinkRecommendations` for enwiki (T386867) [production]
22:00 <sbassett> updated security patch for T355073 (scap update-patch) [production]
19:28 <dancy@deploy1003> Installation of scap version "4.184.1" completed for 2 hosts [production]
19:26 <dancy@deploy1003> Installing scap version "4.184.1" for 2 host(s) [production]
18:14 <fceratto@deploy1003> helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [production]
18:12 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46 [tools]
17:58 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-14, tools-k8s-worker-nfs-46 [tools]
17:46 <jhancock@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:45 <jhancock@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:41 <jhancock@cumin1003> START - Cookbook sre.hosts.provision for host cp2043.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
17:41 <jhancock@cumin1003> START - Cookbook sre.hosts.provision for host sretest2005.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:51 <jhancock@cumin1003> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host sretest2003.codfw.wmnet with OS bookworm [production]
16:45 <jhancock@cumin1003> START - Cookbook sre.hosts.reimage for host sretest2004.codfw.wmnet with OS bookworm [production]
16:42 <volans> uploaded debmonitor-server,python3-debmonitor_0.6.2 to apt.wikimedia.org bookworm-wikimedia [production]
16:40 <ayounsi@cumin2002> END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2009.codfw.wmnet with OS bookworm [production]
16:33 <jhancock@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
16:32 <jhancock@cumin1003> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest2003.codfw.wmnet with reason: host reimage [production]
16:27 <ayounsi@cumin2002> START - Cookbook sre.hosts.reimage for host sretest2009.codfw.wmnet with OS bookworm [production]
16:27 <jhancock@cumin1003> START - Cookbook sre.hosts.downtime for 2:00:00 on sretest2003.codfw.wmnet with reason: host reimage [production]
16:11 <jhancock@cumin1003> START - Cookbook sre.hosts.provision for host sretest2004.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
16:09 <jhancock@cumin1003> START - Cookbook sre.hosts.reimage for host sretest2003.codfw.wmnet with OS bookworm [production]
16:09 <jhancock@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
16:05 <stevemunene@cumin1003> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts an-conf1003.eqiad.wmnet [production]
16:05 <stevemunene@cumin1003> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:05 <stevemunene@cumin1003> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-conf1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1003" [production]
16:05 <stevemunene@cumin1003> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: an-conf1003.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - stevemunene@cumin1003" [production]
16:04 <jhancock@cumin1003> START - Cookbook sre.hosts.provision for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
16:02 <jhancock@cumin1003> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp2055.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
16:02 <fceratto@deploy1003> helmfile [aux-k8s-eqiad] 'sync' command on namespace 'zarcillo' for release 'main' . [production]
16:02 <jhancock@cumin1003> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2003.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
16:01 <stevemunene@cumin1003> START - Cookbook sre.dns.netbox [production]
16:01 <jhancock@cumin1003> START - Cookbook sre.hosts.provision for host cp2055.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]