Search

On or before date

go

2026-07-23 §
12:10	<taavi@cloudcumin1001>	END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance metricsinfra-prometheus-3	[metricsinfra]
12:09	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.remove_instance for instance metricsinfra-prometheus-3	[metricsinfra]
12:05	<taavi@cloudcumin1001>	END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
12:04	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
12:03	<taavi@cloudcumin1001>	END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
12:02	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-prometheus-5.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
11:22	<taavi@cloudcumin1001>	END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance metricsinfra-prometheus-2	[metricsinfra]
11:22	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.remove_instance for instance metricsinfra-prometheus-2	[metricsinfra]
11:21	<taavi@cloudcumin1001>	END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance metricsinfra-controller-2	[metricsinfra]
11:20	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.remove_instance for instance metricsinfra-controller-2	[metricsinfra]
2026-07-22 §
14:47	<taavi@cloudcumin1001>	END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on metricsinfra-prometheus-4.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
14:46	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-prometheus-4.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
14:40	<taavi@cloudcumin1001>	END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) by 8 cores, 16384 ram	[metricsinfra]
14:40	<taavi@cloudcumin1001>	START - Cookbook wmcs.openstack.quota_increase by 8 cores, 16384 ram	[metricsinfra]
2026-07-09 §
13:42	<taavi@cloudcumin1001>	END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on metricsinfra-controller-3.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
13:40	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-controller-3.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
13:30	<taavi@cloudcumin1001>	END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on metricsinfra-controller-3.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
13:29	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-controller-3.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
2026-06-02 §
11:20	<taavi>	renew puppet CA certificate T427905	[metricsinfra]
2026-03-03 §
08:50	<godog>	downgrade grafana to 12.3.4 to work around mariadb compat - T418831	[metricsinfra]
2026-01-12 §
14:25	<taavi>	exclude tools/toolsbeta from generic NfsAlmostFull alert as those projects have a more specific alert with a specific runbook	[metricsinfra]
2025-12-17 §
09:41	<godog>	free up some space on /srv on metricsinfra-prometheus-3	[metricsinfra]
2025-11-18 §
14:41	<taavi@cloudcumin1001>	END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm metricsinfra-thanos-fe-2 (cluster eqiad1)	[metricsinfra]
14:41	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.instance.stop_start vm metricsinfra-thanos-fe-2 (cluster eqiad1)	[metricsinfra]
14:40	<taavi@cloudcumin1001>	END (PASS) - Cookbook wmcs.vps.instance.stop_start (exit_code=0) vm metricsinfra-grafana-2 (cluster eqiad1)	[metricsinfra]
14:39	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.instance.stop_start vm metricsinfra-grafana-2 (cluster eqiad1)	[metricsinfra]
2025-11-04 §
17:38	<dcaro>	removed tools and toolsbeta redis from the scrapes targets (moved to tools/toolsbeta prometheus)	[metricsinfra]
2025-09-26 §
11:40	<godog>	set default timezone for grafana 'wikimedia cloud services' org to UTC	[metricsinfra]
2025-08-27 §
14:06	<dcaro>	remove osbpo repos as they don't work anymore	[metricsinfra]
2025-08-14 §
11:24	<taavi@cloudcumin1001>	END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on metricsinfra-thanos-fe-2.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
11:22	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-thanos-fe-2.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
2025-08-04 §
13:06	<filippo@cloudcumin1001>	END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'filippo' in role 'member' (T401091)	[metricsinfra]
13:06	<filippo@cloudcumin1001>	START - Cookbook wmcs.vps.add_user_to_project for user 'filippo' in role 'member' (T401091)	[metricsinfra]
2025-06-30 §
11:45	<dcaro>	added a new global alert when nfs space is >90%	[metricsinfra]
2025-05-12 §
10:44	<taavi>	add generic TargetDown rule for better detection of issues like T392889	[metricsinfra]
2025-05-09 §
08:47	<taavi>	failing over grafana to grafana-2 T393735	[metricsinfra]
08:44	<taavi@cloudcumin1001>	END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on metricsinfra-grafana-2.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
08:43	<taavi@cloudcumin1001>	START - Cookbook wmcs.vps.refresh_puppet_certs on metricsinfra-grafana-2.metricsinfra.eqiad1.wikimedia.cloud	[metricsinfra]
2025-04-29 §
14:10	<taavi>	upgrading thanos to 0.38 T383966	[metricsinfra]
2025-04-23 §
09:43	<taavi>	updating security group rules to include IPv6 terms	[metricsinfra]
2025-01-31 §
11:38	<dhinus>	rebooting VM metricsinfra-prometheus-3 T385262	[metricsinfra]
11:30	<dhinus>	systemctl restart prometheus@cloud on metricsinfra-prometheus-2 T385262	[metricsinfra]
11:16	<dhinus>	systemctl restart prometheus@cloud on metricsinfra-prometheus-3 T385262	[metricsinfra]
2025-01-20 §
13:01	<dcaro>	stopping and starting metricsinfra-alertmanager-3 to try to get the right network	[metricsinfra]
2024-06-24 §
20:09	<andrew@cloudcumin1001>	END (FAIL) - Cookbook wmcs.openstack.migrate_project_to_ovs (exit_code=1)	[metricsinfra]
19:56	<andrew@cloudcumin1001>	START - Cookbook wmcs.openstack.migrate_project_to_ovs	[metricsinfra]
2024-03-13 §
12:14	<taavi>	MariaDB [prometheusconfig]> delete from alerts where name = 'GridQueueProblem'; # T314664	[metricsinfra]
2023-11-30 §
18:53	<taavi>	no longer send quarry alerts to cloud services team	[metricsinfra]
2023-11-18 §
14:09	<taavi>	reboot metricsinfra-alertmanager-1 to see if it stops flapping a puppet alert	[metricsinfra]
2023-09-29 §
08:24	<wm-bot2>	dcaro@urcuchillay END (PASS) - Cookbook wmcs.openstack.cloudvirt.vm_console (exit_code=0)	[metricsinfra]