| 2024-01-30
      
      § | 
    
  | 19:13 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 19:12 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 19:12 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-8.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 19:04 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 19:04 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8 | [tools] | 
            
  | 19:03 | <taavi@cloudcumin1001> | START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8 | [tools] | 
            
  | 18:51 | <taavi@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 18:48 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 18:48 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-k8s-worker-nfs-8 | [tools] | 
            
  | 18:47 | <taavi@cloudcumin1001> | START - Cookbook wmcs.vps.remove_instance for instance tools-k8s-worker-nfs-8 | [tools] | 
            
  | 18:46 | <taavi@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 18:42 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 18:41 | <taavi@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 18:41 | <taavi@cloudcumin1001> | Added a new k8s worker-nfs tools-k8s-worker-nfs-7.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 18:33 | <taavi@cloudcumin1001> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 18:29 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-41 | [tools] | 
            
  | 18:29 | <andrew@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-41 | [tools] | 
            
  | 18:24 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-40 | [tools] | 
            
  | 18:23 | <andrew@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-40 | [tools] | 
            
  | 18:22 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-39 | [tools] | 
            
  | 18:22 | <andrew@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-39 | [tools] | 
            
  | 18:18 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-38 | [tools] | 
            
  | 18:17 | <andrew@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-38 | [tools] | 
            
  | 18:09 | <andrew@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-37 | [tools] | 
            
  | 18:08 | <andrew@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-37 | [tools] | 
            
  | 15:16 | <dcaro> | restart harbor now that the db is clean (T356037) | [tools] | 
            
  | 15:14 | <dcaro> | restart harbor now that the db is clean (T3543) | [tools] | 
            
  | 13:08 | <taavi> | create no-op DMARC record T354112 | [tools] | 
            
  | 12:39 | <dcaro> | rebuilding all the toolforge images (T354320) | [tools] | 
            
  | 10:16 | <dcaro> | restarting harbor and flushing redis to regenerate cache data (T356037) | [tools] | 
            
  | 09:33 | <dcaro> | cleaning up old schedules on harbor (T356037) | [tools] | 
            
  
    | 2024-01-29
      
      § | 
    
  | 19:46 | <andrew@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36 | [tools] | 
            
  | 19:46 | <andrew@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=99) for host tools-k8s-worker-36 | [tools] | 
            
  | 19:46 | <andrew@cloudcumin1001> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-36 | [tools] | 
            
  | 14:36 | <wmbot~taavi@runko> | END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-mail-4.tools.eqiad1.wikimedia.cloud | [tools] | 
            
  | 14:34 | <wmbot~taavi@runko> | START - Cookbook wmcs.vps.refresh_puppet_certs on tools-mail-4.tools.eqiad1.wikimedia.cloud | [tools] | 
            
  | 12:06 | <wmbot~taavi@runko> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 12:06 | <wmbot~taavi@runko> | Added a new k8s worker-nfs tools-k8s-worker-nfs-6.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 11:55 | <wmbot~taavi@runko> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 11:51 | <wmbot~taavi@runko> | END (FAIL) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=99) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 11:51 | <wmbot~taavi@runko> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 11:37 | <wmbot~taavi@runko> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 11:37 | <wmbot~taavi@runko> | Added a new k8s worker-nfs tools-k8s-worker-nfs-5.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 11:26 | <wmbot~taavi@runko> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 11:23 | <wmbot~taavi@runko> | END (PASS) - Cookbook wmcs.toolforge.add_k8s_node (exit_code=0) for a worker-nfs role in the tools cluster | [tools] | 
            
  | 11:22 | <wmbot~taavi@runko> | Added a new k8s worker-nfs tools-k8s-worker-nfs-4.tools.eqiad1.wikimedia.cloud to the cluster | [tools] | 
            
  | 11:12 | <wmbot~taavi@runko> | START - Cookbook wmcs.toolforge.add_k8s_node for a worker-nfs role in the tools cluster | [tools] | 
            
  | 11:12 | <wmbot~taavi@runko> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-35 | [tools] | 
            
  | 11:10 | <wmbot~taavi@runko> | START - Cookbook wmcs.toolforge.remove_k8s_node for host tools-k8s-worker-35 | [tools] | 
            
  | 11:10 | <wmbot~taavi@runko> | END (PASS) - Cookbook wmcs.toolforge.remove_k8s_node (exit_code=0) for host tools-k8s-worker-34 | [tools] |