| 2024-02-22
      
      § | 
    
  | 12:57 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' (T319184) | [admin] | 
            
  | 12:53 | <aborrero@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=99) on host 'cloudvirt1034.eqiad.wmnet' (T319184) | [admin] | 
            
  | 12:32 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1034.eqiad.wmnet' (T319184) | [admin] | 
            
  | 11:55 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) | [admin] | 
            
  | 11:54 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance | [admin] | 
            
  | 09:01 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) | [admin] | 
            
  | 09:00 | <taavi@cloudcumin1001> | START - Cookbook wmcs.openstack.restart_openstack | [admin] | 
            
  
    | 2024-02-21
      
      § | 
    
  | 13:44 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.set_maintenance (exit_code=0) (T319184) | [admin] | 
            
  | 13:43 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.set_maintenance (T319184) | [admin] | 
            
  | 13:20 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) (T319184) | [admin] | 
            
  | 13:19 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (T319184) | [admin] | 
            
  | 12:50 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' (T319184) | [admin] | 
            
  | 12:50 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' (T319184) | [admin] | 
            
  | 12:03 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1033.eqiad.wmnet' (T319184) | [admin] | 
            
  | 11:44 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1033.eqiad.wmnet' (T319184) | [admin] | 
            
  
    | 2024-02-20
      
      § | 
    
  | 11:45 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) | [admin] | 
            
  | 11:45 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance | [admin] | 
            
  | 11:45 | <aborrero@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) | [admin] | 
            
  | 11:45 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance | [admin] | 
            
  | 11:30 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=0) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184) | [admin] | 
            
  | 11:30 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184) | [admin] | 
            
  
    | 2024-02-19
      
      § | 
    
  | 12:33 | <aborrero@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184) | [admin] | 
            
  | 12:32 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184) | [admin] | 
            
  | 12:02 | <aborrero@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184) | [admin] | 
            
  | 12:02 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184) | [admin] | 
            
  | 12:00 | <aborrero@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.cloudvirt.post-reimage (exit_code=99) preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184) | [admin] | 
            
  | 12:00 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.post-reimage preparing cloudvirt cloudvirt1032.eqiad.wmnet for duty (nova discovery, canary VM) Pending aggregates though. (T319184) | [admin] | 
            
  | 10:09 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=0) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184) | [admin] | 
            
  | 09:52 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184) | [admin] | 
            
  | 09:49 | <aborrero@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.cloudvirt.pre-reimage (exit_code=99) prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184) | [admin] | 
            
  | 09:49 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.pre-reimage prepare cloudvirt1032.eqiad.wmnet for reimage (drain, remove nova agent, etc) (T319184) | [admin] | 
            
  
    | 2024-02-15
      
      § | 
    
  | 17:34 | <wmbot~fran@wmf3169> | START - Cookbook wmcs.openstack.roll_reboot_cloudnets (T356975) | [admin] | 
            
  | 15:34 | <taavi> | restart radosgw in eqiad as I am seeing 500 errors | [admin] | 
            
  | 14:22 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=0) | [admin] | 
            
  | 14:22 | <taavi@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance | [admin] | 
            
  | 14:22 | <taavi@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.cloudvirt.unset_maintenance (exit_code=99) | [admin] | 
            
  | 14:22 | <taavi@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.unset_maintenance | [admin] | 
            
  | 12:05 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.drain (exit_code=0) on host 'cloudvirt1031.eqiad.wmnet' (T319184) | [admin] | 
            
  | 11:58 | <dhinus> | restore correct aggregate "localdisk" for cloudvirtlocal1001 and remove "maintenance" | [admin] | 
            
  | 11:52 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.drain on host 'cloudvirt1031.eqiad.wmnet' (T319184) | [admin] | 
            
  | 05:51 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1044.eqiad.wmnet}' | [admin] | 
            
  | 05:41 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1045.eqiad.wmnet}' | [admin] | 
            
  | 05:19 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1044.eqiad.wmnet}' | [admin] | 
            
  | 05:18 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1047.eqiad.wmnet}' | [admin] | 
            
  | 05:00 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1046.eqiad.wmnet}' | [admin] | 
            
  | 04:52 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1047.eqiad.wmnet}' | [admin] | 
            
  | 04:52 | <andrew@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=99) on hosts matched by 'D{cloudvirt1046.eqiad.wmnet[B}' | [admin] | 
            
  | 04:51 | <andrew@cloudcumin1001> | START - Cookbook wmcs.openstack.cloudvirt.safe_reboot on hosts matched by 'D{cloudvirt1046.eqiad.wmnet[B}' | [admin] | 
            
  | 04:50 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1049.eqiad.wmnet}' | [admin] | 
            
  | 04:48 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.cloudvirt.safe_reboot (exit_code=0) on hosts matched by 'D{cloudvirt1048.eqiad.wmnet}' | [admin] |