| 2024-09-19
      
      ยง | 
    
  | 10:38 | <brouberol@deploy1003> | helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'. | [production] | 
            
  | 10:30 | <elukey@deploy1003> | Finished scap sync-world: Remove network policies for old poolcounter nodes. (duration: 08m 55s) | [production] | 
            
  | 10:26 | <jiji@deploy1003> | helmfile [eqiad] DONE helmfile.d/services/ipoid: apply | [production] | 
            
  | 10:26 | <jiji@deploy1003> | helmfile [eqiad] START helmfile.d/services/ipoid: apply | [production] | 
            
  | 10:26 | <XioNoX> | enable gNMI on cloudsw | [production] | 
            
  | 10:26 | <jiji@deploy1003> | helmfile [eqiad] DONE helmfile.d/services/ipoid: apply | [production] | 
            
  | 10:26 | <jiji@deploy1003> | helmfile [eqiad] START helmfile.d/services/ipoid: apply | [production] | 
            
  | 10:23 | <moritzm> | rebalance ganeti group C following the various switch maintenances T370630 | [production] | 
            
  | 10:21 | <elukey@deploy1003> | Started scap sync-world: Remove network policies for old poolcounter nodes. | [production] | 
            
  | 10:03 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51 | [admin] | 
            
  | 10:03 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/51 | [admin] | 
            
  | 09:56 | <wmbot~dcaro@urcuchillay> | END (ERROR) - Cookbook wmcs.ceph.osd.undrain_node (exit_code=97) (T374043) | [admin] | 
            
  | 09:55 | <wmbot~dcaro@urcuchillay> | START - Cookbook wmcs.ceph.osd.undrain_node (T374043) | [admin] | 
            
  | 09:51 | <arturo> | [codfw1dev] play with neutron default security group rules (delete, create them, etc) T375111 | [admin] | 
            
  | 09:48 | <volans> | uploaded python3-wmflib_1.2.7 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia,bookworm-wikimedia | [production] | 
            
  | 09:44 | <vgutierrez> | testing purged 0.24 in cp4038 - T334078 | [production] | 
            
  | 09:44 | <jiji@deploy1003> | helmfile [eqiad] DONE helmfile.d/services/ipoid: apply | [production] | 
            
  | 09:44 | <jiji@deploy1003> | helmfile [eqiad] START helmfile.d/services/ipoid: apply | [production] | 
            
  | 09:42 | <jiji@deploy1003> | helmfile [eqiad] DONE helmfile.d/services/ipoid: apply | [production] | 
            
  | 09:42 | <jiji@deploy1003> | helmfile [eqiad] START helmfile.d/services/ipoid: apply | [production] | 
            
  | 09:42 | <jiji@deploy1003> | helmfile [codfw] DONE helmfile.d/services/ipoid: apply | [production] | 
            
  | 09:41 | <jiji@deploy1003> | helmfile [codfw] START helmfile.d/services/ipoid: apply | [production] | 
            
  | 09:27 | <aborrero@cloudcumin1001> | END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | [admin] | 
            
  | 09:27 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | [admin] | 
            
  | 09:25 | <aborrero@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | [admin] | 
            
  | 09:25 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | [admin] | 
            
  | 09:24 | <btullis> | rebooting analytics[1074-1075] with batch size 2 for T366555 | [analytics] | 
            
  | 09:23 | <arnaudb@cumin1002> | END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1246.eqiad.wmnet with OS bookworm | [production] | 
            
  | 09:23 | <btullis@cumin1002> | START - Cookbook sre.hadoop.reboot-workers for Hadoop analytics cluster | [production] | 
            
  | 09:11 | <aborrero@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | [admin] | 
            
  | 09:10 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | [admin] | 
            
  | 09:10 | <arnaudb@cumin1002> | START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm | [production] | 
            
  | 09:10 | <arnaudb@cumin1002> | END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1246.eqiad.wmnet with OS bookworm | [production] | 
            
  | 09:08 | <aborrero@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | [admin] | 
            
  | 09:07 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | [admin] | 
            
  | 09:07 | <aborrero@cloudcumin1001> | END (FAIL) - Cookbook wmcs.openstack.tofu (exit_code=99) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | [admin] | 
            
  | 09:07 | <aborrero@cloudcumin1001> | START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/50 | [admin] | 
            
  | 09:03 | <hashar> | Cleared deprecated approvals from CI Jenkins # T375160 | [releng] | 
            
  | 09:02 | <hashar> | CI Jenkins: approved 3 scripts we wrote which were pending approval at https://integration.wikimedia.org/ci/manage/scriptApproval/ # T375160 | [releng] | 
            
  | 09:01 | <hashar> | CI Jenkins: approved 3 scripts we wrote which were pending approval at https://integration.wikimedia.org/ci/manage/scriptApproval/ | [releng] | 
            
  | 08:51 | <arnaudb@cumin1002> | START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm | [production] | 
            
  | 08:50 | <arnaudb@cumin1002> | END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db1246.eqiad.wmnet with OS bookworm | [production] | 
            
  | 08:45 | <hashar> | Restarting CI Jenkins with Java 17 # T359795 | [production] | 
            
  | 08:31 | <arnaudb@cumin1002> | START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm | [production] | 
            
  | 08:31 | <_joe_> | deployed conftool 3.2.4 T375059 T373449 | [production] | 
            
  | 08:30 | <ayounsi@cumin1002> | END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox | [production] | 
            
  | 08:29 | <ayounsi@cumin1002> | START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox | [production] | 
            
  | 08:16 | <jnuche@deploy1003> | rebuilt and synchronized wikiversions files: group2 to 1.43.0-wmf.23  refs T373642 | [production] | 
            
  | 08:04 | <ayounsi@cumin1002> | END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary | [production] | 
            
  | 08:04 | <ayounsi@cumin1002> | START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary | [production] |