| 2024-11-04
      
      ยง | 
    
  | 13:09 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 13:08 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-12 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 13:08 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 13:07 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-11 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 13:07 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 13:06 | <Dreamy_Jazz> | Started MediaModeration scan on all wikis other than s4 (commonswiki + testcommonswiki) - https://wikitech.wikimedia.org/wiki/MediaModeration | [production] | 
            
  | 13:06 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 13:04 | <sstefanova@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component builds-api | [tools] | 
            
  | 13:04 | <sstefanova@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component builds-api | [tools] | 
            
  | 13:02 | <sstefanova@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component builds-api | [tools] | 
            
  | 12:55 | <ladsgroup@cumin1002> | dbctl commit (dc=all): 'Repooling after maintenance db1167 (T376905)', diff saved to https://phabricator.wikimedia.org/P70861 and previous config saved to /var/cache/conftool/dbconfig/20241104-125459-ladsgroup.json | [production] | 
            
  | 12:55 | <raymond-ndibe@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:49 | <XioNoX> | deploy "Add temporary LVS community for liberica test" - T378453 | [production] | 
            
  | 12:49 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:47 | <raymond-ndibe@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=99) for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:45 | <ladsgroup@cumin1002> | dbctl commit (dc=all): 'Depooling db1167 (T376905)', diff saved to https://phabricator.wikimedia.org/P70860 and previous config saved to /var/cache/conftool/dbconfig/20241104-124533-ladsgroup.json | [production] | 
            
  | 12:45 | <ladsgroup@cumin1002> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance | [production] | 
            
  | 12:45 | <ladsgroup@cumin1002> | START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on an-redacteddb1001.eqiad.wmnet,clouddb[1016,1020].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance | [production] | 
            
  | 12:45 | <ladsgroup@cumin1002> | END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance | [production] | 
            
  | 12:44 | <ladsgroup@cumin1002> | START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance | [production] | 
            
  | 12:44 | <sstefanova@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-api | [toolsbeta] | 
            
  | 12:41 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-10 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:41 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:40 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-nfs-1 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:40 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:39 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-108 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:39 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:38 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-107 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:38 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:38 | <sstefanova@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component builds-api | [toolsbeta] | 
            
  | 12:36 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-106 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:36 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:35 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-105 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:35 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:35 | <elukey@cumin1002> | END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART | [production] | 
            
  | 12:34 | <isaranto@deploy2002> | helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' . | [production] | 
            
  | 12:34 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-103 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:34 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:33 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-worker-102 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:24 | <elukey@cumin1002> | START - Cookbook sre.hosts.provision for host ganeti1052.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART | [production] | 
            
  | 12:22 | <raymond-ndibe@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.k8s.worker.upgrade (exit_code=0) for node tools-k8s-control-9 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:22 | <isaranto@deploy2002> | helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' . | [production] | 
            
  | 12:22 | <sstefanova@cloudcumin1001> | END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component envvars-api | [tools] | 
            
  | 12:22 | <isaranto@deploy2002> | helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' . | [production] | 
            
  | 12:20 | <isaranto@deploy2002> | helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' . | [production] | 
            
  | 12:19 | <isaranto@deploy2002> | helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' . | [production] | 
            
  | 12:19 | <isaranto@deploy2002> | helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' . | [production] | 
            
  | 12:16 | <sstefanova@cloudcumin1001> | START - Cookbook wmcs.toolforge.component.deploy for component envvars-api | [tools] | 
            
  | 12:13 | <raymond-ndibe@cloudcumin1001> | START - Cookbook wmcs.toolforge.k8s.worker.upgrade for node tools-k8s-control-9 from 1.27.16 to 1.28.14 (T362867) | [tools] | 
            
  | 12:11 | <sstefanova@cloudcumin1001> | END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component envvars-api | [tools] |