| 
      
        2023-02-27
      
      §
     | 
  
    
  | 18:41 | 
  <wm-bot2> | 
  Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 18:38 | 
  <wm-bot2> | 
  Rebooting node cloudcephosd1004.eqiad.wmnet (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 18:38 | 
  <wm-bot2> | 
  Adding OSD cloudcephosd1004.eqiad.wmnet... (1/1) (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 18:38 | 
  <wm-bot2> | 
  Adding new OSDs ['cloudcephosd1004.eqiad.wmnet'] to the cluster (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 18:14 | 
  <wm-bot2> | 
  The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1003.eqiad.wmnet'] (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:13 | 
  <wm-bot2> | 
  Added 1 new OSDs ['cloudcephosd1003.eqiad.wmnet'] (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:13 | 
  <wm-bot2> | 
  Added OSD cloudcephosd1003.eqiad.wmnet... (1/1) (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:05 | 
  <wm-bot2> | 
  Finished rebooting node cloudcephosd1003.eqiad.wmnet (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:02 | 
  <wm-bot2> | 
  Rebooting node cloudcephosd1003.eqiad.wmnet (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:01 | 
  <wm-bot2> | 
  Adding OSD cloudcephosd1003.eqiad.wmnet... (1/1) (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:01 | 
  <wm-bot2> | 
  Adding new OSDs ['cloudcephosd1003.eqiad.wmnet'] to the cluster (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 15:59 | 
  <wm-bot2> | 
  Adding OSD coludcephosd1003.eqiad.wmnet... (1/1) (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 15:59 | 
  <wm-bot2> | 
  Adding new OSDs ['coludcephosd1003.eqiad.wmnet'] to the cluster (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 15:58 | 
  <wm-bot2> | 
  Adding OSD coludcephosd1003... (1/1) (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 15:58 | 
  <wm-bot2> | 
  Adding new OSDs ['coludcephosd1003'] to the cluster (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  
    | 
      
        2023-02-22
      
      §
     | 
  
    
  | 08:25 | 
  <wm-bot2> | 
  Depooled and destroyed OSD daemons [31, 30, 29, 28, 27, 26, 25, 24] and removed the OSD host cloudcephosd1004 from the CRUSH map. (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 08:25 | 
  <wm-bot2> | 
  Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 08:22 | 
  <wm-bot2> | 
  Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 08:15 | 
  <wm-bot2> | 
  Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 08:13 | 
  <wm-bot2> | 
  Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 07:23 | 
  <wm-bot2> | 
  Destroying OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 07:02 | 
  <wm-bot2> | 
  Depooling OSDs with ids in [31, 30, 29, 28, 27, 26, 25, 24] on cloudcephosd1004 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  
    | 
      
        2023-02-21
      
      §
     | 
  
    
  | 20:27 | 
  <andrewbogott> | 
  deleted 200 more orphaned VM images with wmcs-novastats-cephleaks | 
  [admin] | 
            
  | 17:18 | 
  <andrewbogott> | 
  shutting down postgres on clouddb1004/1003, then shutting down the vms | 
  [admin] | 
            
  | 15:50 | 
  <wm-bot2> | 
  Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 15:48 | 
  <wm-bot2> | 
  Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 14:21 | 
  <wm-bot2> | 
  Destroying OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 14:00 | 
  <wm-bot2> | 
  Depooling OSDs with ids in [71, 70, 69, 68, 67, 66, 65, 64] on cloudcephosd1003 from eqiad1 (T329502) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  
    | 
      
        2023-02-16
      
      §
     | 
  
    
  | 19:13 | 
  <wm-bot2> | 
  The cluster is now rebalanced after adding the new OSDs ['cloudcephosd1002.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 17:55 | 
  <dcaro> | 
  Manually zapped /dev/sdc on cloudcephosd1002, probably a leftover drive since the beginning (or during the reimage the drives changed names, and this one had leftovers from the previous OS) (T329498) | 
  [admin] | 
            
  | 17:47 | 
  <wm-bot2> | 
  Added 1 new OSDs ['cloudcephosd1002.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 17:47 | 
  <wm-bot2> | 
  Added OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 17:42 | 
  <wm-bot2> | 
  Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 17:41 | 
  <wm-bot2> | 
  Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:50 | 
  <wm-bot2> | 
  Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:50 | 
  <wm-bot2> | 
  Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:01 | 
  <wm-bot2> | 
  Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:01 | 
  <wm-bot2> | 
  Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:00 | 
  <wm-bot2> | 
  Adding OSD cloudcephosd1002.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 16:00 | 
  <wm-bot2> | 
  Adding new OSDs ['cloudcephosd1002.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 14:05 | 
  <wm-bot2> | 
  Added 1 new OSDs ['cloudcephosd1001.eqiad.wmnet'] (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 13:29 | 
  <wm-bot2> | 
  Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 13:29 | 
  <wm-bot2> | 
  Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 13:24 | 
  <wm-bot2> | 
  Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 13:24 | 
  <wm-bot2> | 
  Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 13:23 | 
  <wm-bot2> | 
  Destroying OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 13:21 | 
  <wm-bot2> | 
  Depooling OSDs with ids in [63, 62, 61, 60, 59, 58, 57, 56] on cloudcephosd1002 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 13:14 | 
  <wm-bot2> | 
  Adding OSD cloudcephosd1001.eqiad.wmnet... (1/1) (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 13:14 | 
  <wm-bot2> | 
  Adding new OSDs ['cloudcephosd1001.eqiad.wmnet'] to the cluster (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] | 
            
  | 11:20 | 
  <wm-bot2> | 
  Destroying OSDs with ids in [53, 52, 51, 50] on cloudcephosd1001 from eqiad1 (T329498) - cookbook ran by dcaro@vulcanus | 
  [admin] |