1-50 of 3683 results (36ms)
2023-02-21 §
20:27 <andrewbogott> deleted 200 more orphaned VM images with wmcs-novastats-cephleaks [admin]
17:18 <andrewbogott> shutting down postgres on clouddb1004/1003, then shutting down the vms [admin]
15:50 <wm-bot2> Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus [admin]
15:48 <wm-bot2> Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus [admin]
14:21 <wm-bot2> Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus [admin]
14:00 <wm-bot2> Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus [admin]
2023-02-16 §
19:13 <wm-bot2> The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1002.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus [admin]
17:55 <dcaro> Manually zapped /dev/sdc on cloudcephosd1002, probably a leftover drive since the beginning (or during the reimage the drives changed names, and this one had leftovers from the previous OS) (T329498) [admin]
17:47 <wm-bot2> Added 1 new OSDs ['cloudcephosd1002.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus [admin]
17:47 <wm-bot2> Added OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus [admin]
17:42 <wm-bot2> Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus [admin]
17:41 <wm-bot2> Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus [admin]
16:50 <wm-bot2> Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus [admin]
16:50 <wm-bot2> Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus [admin]
16:01 <wm-bot2> Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus [admin]
16:01 <wm-bot2> Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus [admin]
16:00 <wm-bot2> Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus [admin]
16:00 <wm-bot2> Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus [admin]
14:05 <wm-bot2> Added 1 new OSDs ['cloudcephosd1001.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus [admin]
13:29 <wm-bot2> Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus [admin]
13:29 <wm-bot2> Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus [admin]
13:24 <wm-bot2> Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus [admin]
13:24 <wm-bot2> Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus [admin]
13:23 <wm-bot2> Destroying OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus [admin]
13:21 <wm-bot2> Depooling OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus [admin]
13:14 <wm-bot2> Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus [admin]
13:14 <wm-bot2> Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus [admin]
11:20 <wm-bot2> Destroying OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus [admin]
11:19 <wm-bot2> Depooling OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus [admin]
11:03 <wm-bot2> Destroying OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus [admin]
11:01 <wm-bot2> Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus [admin]
10:59 <wm-bot2> Depooling OSDs with ids in [55, 54, 53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus [admin]
10:15 <dcaro> purges osd daemons 48 and 40 from eqiad ceph cluster (T329709) [admin]
2023-02-15 §
14:53 <andrewbogott> deleting another 100 leaked VM images with wmcs-novastats-cephleaks [admin]
13:39 <wm-bot2> Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus [admin]
13:14 <wm-bot2> Destroying OSDs with id [48] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus [admin]
13:13 <wm-bot2> Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus [admin]
13:11 <wm-bot2> Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus [admin]
13:11 <wm-bot2> Destroying OSDs with id [12345] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus [admin]
13:10 <wm-bot2> Destroying OSDs with id [[12345]] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus [admin]
13:09 <wm-bot2> Destroying OSDs with id ['12345'] on cloudcephosd1001 from eqiad1 - cookbook ran by dcaro@vulcanus [admin]
2023-02-14 §
13:17 <andrewbogott> restarting all eqiad1 openstack services because that seems to sometimes help things *shrug* [admin]
2023-02-13 §
14:06 <wm-bot2> Set the ceph cluster for eqiad1 in maintenance, alert silence ids: 8fbf6bfd-eec1-4d81-8e0d-ea431d8411ee (T329498) - cookbook ran by dcaro@vulcanus [admin]
13:32 <taavi> re-enable puppet on labstore1004 T329377 [admin]
2023-02-09 §
21:17 <andrewbogott> deleted 10% of leaked VM ceph images using wmcs-novastats-cephleaks (only 10% out of an abundance of caution) [admin]
2023-02-08 §
17:08 <arturo> changing to cloudgw network setup, make VIPs /32 (T295774) [admin]
2023-02-07 §
11:26 <arturo> [codfw1dev] testing network changes in cloudgw, expect unrealiable network (T295774) [admin]
2023-02-04 §
13:44 <taavi> drop old columns from oathauth_users table on labtestwiki T328131 [admin]
2023-02-03 §
15:00 <andrewbogott> restarted nova services in eqiad1 in an attempt to eke out another day or two of stability [admin]
14:13 <taavi> attached GrapheSuppression developer account to wikitech [admin]