1-50 of 10000 results (43ms)
2026-06-16 §
17:51 <andrewbogott> ceph tell osd.126,127,129,131 compact [admin]
17:04 <andrewbogott> rebooting cloudcephosd1037 and cloudcephosd1038 because of missing volumes. The volumes reappeared after boot. [admin]
16:47 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.upgrade_osds (exit_code=99) (T428385) [admin]
16:47 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.upgrade_osds (T428385) [admin]
16:47 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' (T429361) [admin]
16:36 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' (T429361) [admin]
16:35 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' (T429361) [admin]
16:23 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' (T429361) [admin]
16:22 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet' (T429361) [admin]
16:12 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' (T429361) [admin]
16:01 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2010-dev.codfw.wmnet' (T429361) [admin]
15:52 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2010-dev.codfw.wmnet' (T429361) [admin]
15:52 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2011-dev.codfw.wmnet' (T429361) [admin]
15:52 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2011-dev.codfw.wmnet' (T429361) [admin]
15:52 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2005-dev.codfw.wmnet' (T429361) [admin]
15:41 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2005-dev.codfw.wmnet' (T429361) [admin]
15:40 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2006-dev.codfw.wmnet' (T429361) [admin]
15:33 <andrewbogott> restarting quite a few other ceph-osd services in an attempt to quiet some slow op alerts [admin]
15:30 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006-dev.codfw.wmnet' (T429361) [admin]
15:13 <andrewbogott> systemctl restart ceph-osd@58.service due to slow ops [admin]
14:57 <andrewbogott> "systemctl restart ceph-osd@281.service" as 281 shows as down [admin]
14:55 <andrewbogott> "systemctl restart ceph-osd@289.service" as 289 shows as down [admin]
13:24 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudservices.safe_reboot (exit_code=0) on hosts matched by 'P{O:wmcs::openstack::codfw1dev::services}' [admin]
13:18 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudservices.safe_reboot on hosts matched by 'P{O:wmcs::openstack::codfw1dev::services}' [admin]
2026-06-15 §
21:48 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.upgrade_osds (exit_code=0) [admin]
21:04 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.upgrade_osds (T428385) [admin]
21:02 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.upgrade_osds (exit_code=99) [admin]
19:46 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.upgrade_osds (T428385) [admin]
19:32 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.upgrade_osds (exit_code=0) [admin]
19:00 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.upgrade_osds (T428385) [admin]
18:40 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.upgrade_osds (exit_code=99) [admin]
18:27 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.upgrade_osds (T428385) [admin]
17:07 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.upgrade_osds (exit_code=99) [admin]
14:31 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.upgrade_osds (T428385) [admin]
14:10 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.ceph.upgrade_mons (exit_code=0) [admin]
13:50 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.upgrade_mons (T428385) [admin]
13:48 <andrew@cloudcumin1001> END (ERROR) - Cookbook wmcs.ceph.upgrade_osds (exit_code=97) [admin]
13:38 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.upgrade_osds (T428385) [admin]
13:08 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.ceph.upgrade_mons (exit_code=99) [admin]
12:46 <andrew@cloudcumin1001> START - Cookbook wmcs.ceph.upgrade_mons (T428385) [admin]
2026-06-12 §
14:00 <volans> clearing cloudback-original snapshots of cinder volumes created in 2026 to allow the backups to not fail for T428995 [admin]
2026-06-11 §
17:53 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1011.eqiad.wmnet' (T428549) [admin]
17:39 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1011.eqiad.wmnet' (T428549) [admin]
17:18 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1007.eqiad.wmnet' (T428549) [admin]
17:05 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1007.eqiad.wmnet' (T428549) [admin]
16:50 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol1006.eqiad.wmnet' (T428549) [admin]
16:37 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol1006.eqiad.wmnet' (T428549) [admin]
16:33 <andrew@cloudcumin1001> END (FAIL) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=99) on host 'cloudcontrol2006.eqiad.wmnet' (T428549) [admin]
16:33 <andrew@cloudcumin1001> START - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node on host 'cloudcontrol2006.eqiad.wmnet' (T428549) [admin]
15:11 <andrew@cloudcumin1001> END (PASS) - Cookbook wmcs.openstack.cloudcontrol.upgrade_openstack_node (exit_code=0) on host 'cloudcontrol2010-dev.codfw.wmnet' (T428549) [admin]