4501-4550 of 10000 results (42ms)
2025-05-12 §
03:15 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services [admin]
03:07 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [admin]
02:36 <andrewbogott> rebooting cloudnet2005-dev from mgmt -- ssh is failing and the console shows a user prompt but not a password prompt. [admin]
02:36 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.k8s.reboot (exit_code=0) for tools-k8s-worker-nfs-19 [tools]
02:32 <andrew@cloudcumin1001> START - Cookbook wmcs.toolforge.k8s.reboot for tools-k8s-worker-nfs-19 [tools]
2025-05-11 §
22:55 <tchin@deploy1003> Finished deploy [airflow-dags/analytics@301c74b]: Deploying airflow artifacts for T384962 (duration: 02m 01s) [production]
22:54 <tchin@deploy1003> Started deploy [airflow-dags/analytics@301c74b]: Deploying airflow artifacts for T384962 [production]
13:42 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment eqiad1 for service: project,nova [admin]
13:37 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.restart_openstack on deployment eqiad1 for service: project,nova [admin]
2025-05-10 §
19:58 <lucaswerkmeister> webservice restart (per request on behalf of tool maintainer, as the bastion is having issues atm) [tools.letaxobot]
17:35 <lucaswerkmeister> root@tools-bastion-13:~# systemctl restart sssd-sudo{,.socket} # looks like the reset-failed didn’t work properly, systemd didn’t even try to start the service again afaict (T393732) [tools]
17:33 <lucaswerkmeister> root@tools-bastion-13:~# systemctl reset-failed sssd-{pam,sudo}.service && systemctl restart sssd-pam{,-priv}.socket # try to reset the rate limits this way (T393732) [tools]
16:22 <lucaswerkmeister> systemctl restart sssd-{pam{,-priv},sudo}.socket # service-start-limit-hit, T393732? [tools]
14:10 <lucaswerkmeister> root@tools-bastion-13:~# systemctl restart sssd-sudo.socket # service-start-limit-hit, T393732? [tools]
11:53 <lucaswerkmeister> T393732 note: restart of sssd-pam.service actually failed, “may be requested by dependency only”; overall it still seems to have worked though (so next time restarting the sockets is probably sufficient) [tools]
11:52 <lucaswerkmeister> root@tools-bastion-13:~# systemctl restart sssd-pam{,{,-priv}.socket} # all three failed with start-limit-hit / Start request repeated too quickly; T393732? [tools]
00:41 <dani@deploy1003> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
00:41 <dani@deploy1003> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
00:41 <dani@deploy1003> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
00:41 <dani@deploy1003> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
00:41 <dani@deploy1003> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
00:41 <dani@deploy1003> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
00:23 <dani@deploy1003> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
00:22 <dani@deploy1003> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
00:22 <dani@deploy1003> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
00:22 <dani@deploy1003> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
00:22 <dani@deploy1003> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
00:22 <dani@deploy1003> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
00:16 <dani@deploy1003> helmfile [codfw] DONE helmfile.d/services/miscweb: apply [production]
00:16 <dani@deploy1003> helmfile [codfw] START helmfile.d/services/miscweb: apply [production]
00:16 <dani@deploy1003> helmfile [eqiad] DONE helmfile.d/services/miscweb: apply [production]
00:15 <dani@deploy1003> helmfile [eqiad] START helmfile.d/services/miscweb: apply [production]
00:15 <dani@deploy1003> helmfile [staging] DONE helmfile.d/services/miscweb: apply [production]
00:15 <dani@deploy1003> helmfile [staging] START helmfile.d/services/miscweb: apply [production]
2025-05-09 §
23:02 <vriley@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-fe1007.eqiad.wmnet with OS bullseye [production]
22:18 <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [toolsbeta]
22:11 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component builds-api [toolsbeta]
22:10 <wmbot~bd808@tools-bastion-12> Deployed container built from8601a403 [tools.gitlab-content]
22:10 <raymond-ndibe@cloudcumin1001> END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api [toolsbeta]
22:10 <wmbot~bd808@tools-bastion-12> Built container from 8601a403 [tools.gitlab-content]
22:10 <vriley@cumin1002> START - Cookbook sre.hosts.reimage for host thanos-fe1007.eqiad.wmnet with OS bullseye [production]
22:09 <vriley@cumin1002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host thanos-fe1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
22:08 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component builds-api [toolsbeta]
22:03 <vriley@cumin1002> START - Cookbook sre.hosts.provision for host thanos-fe1007.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART and with Dell SCP reboot policy FORCED [production]
22:01 <raymond-ndibe@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [toolsbeta]
22:00 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component builds-api [toolsbeta]
21:57 <vriley@cumin1002> END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host thanos-fe1007.eqiad.wmnet with OS bullseye [production]
21:56 <raymond-ndibe@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [toolsbeta]
21:56 <raymond-ndibe@cloudcumin1001> START - Cookbook wmcs.toolforge.component.deploy for component builds-api [toolsbeta]
21:54 <raymond-ndibe@cloudcumin1001> END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api [toolsbeta]