| 2024-09-09
      
      ยง | 
    
  | 20:57 | <kindrobot@deploy1003> | jforrester, toyofuku, kindrobot: Continuing with sync | [production] | 
            
  | 20:51 | <kindrobot@deploy1003> | jforrester, toyofuku, kindrobot: Backport for [[gerrit:1071281|Release donate link to pilot wikis (French Wikipedia and Wikifunctions) (T373585)]], [[gerrit:1071202|Fix typo in browser vendor prefix (T374180)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) | [production] | 
            
  | 20:49 | <kindrobot@deploy1003> | Started scap sync-world: Backport for [[gerrit:1071281|Release donate link to pilot wikis (French Wikipedia and Wikifunctions) (T373585)]], [[gerrit:1071202|Fix typo in browser vendor prefix (T374180)]] | [production] | 
            
  | 19:33 | <jforrester@deploy1003> | Finished scap sync-world: Backport for [[gerrit:1071690|ZObjectStructureValidator::validate: use set_time_limit() to limit in the case of run-away JsonSchema (T374241)]] (duration: 08m 19s) | [production] | 
            
  | 19:28 | <jforrester@deploy1003> | jforrester: Continuing with sync | [production] | 
            
  | 19:27 | <jforrester@deploy1003> | jforrester: Backport for [[gerrit:1071690|ZObjectStructureValidator::validate: use set_time_limit() to limit in the case of run-away JsonSchema (T374241)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) | [production] | 
            
  | 19:25 | <jforrester@deploy1003> | Started scap sync-world: Backport for [[gerrit:1071690|ZObjectStructureValidator::validate: use set_time_limit() to limit in the case of run-away JsonSchema (T374241)]] | [production] | 
            
  | 18:16 | <ejegg> | payments-wiki upgraded from 4f3500f7 to 672c9fb6 | [production] | 
            
  | 17:44 | <jforrester@deploy1003> | Finished scap sync-world: Backport for [[gerrit:1071266|ZObjectStore::findZTesterResult: Trim our own error so we don't break logstash (T374241)]] (duration: 12m 05s) | [production] | 
            
  | 17:39 | <jforrester@deploy1003> | jforrester: Continuing with sync | [production] | 
            
  | 17:33 | <jforrester@deploy1003> | jforrester: Backport for [[gerrit:1071266|ZObjectStore::findZTesterResult: Trim our own error so we don't break logstash (T374241)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) | [production] | 
            
  | 17:31 | <jforrester@deploy1003> | Started scap sync-world: Backport for [[gerrit:1071266|ZObjectStore::findZTesterResult: Trim our own error so we don't break logstash (T374241)]] | [production] | 
            
  | 17:15 | <ebernhardson@deploy1003> | helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply | [production] | 
            
  | 17:15 | <ebernhardson@deploy1003> | helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply | [production] | 
            
  | 17:08 | <ebernhardson@deploy1003> | helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply | [production] | 
            
  | 17:07 | <ebernhardson@deploy1003> | helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply | [production] | 
            
  | 17:05 | <pt1979@cumin2002> | END (PASS) - Cookbook sre.dns.netbox (exit_code=0) | [production] | 
            
  | 17:05 | <pt1979@cumin2002> | END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add nes frack firewalls - pt1979@cumin2002" | [production] | 
            
  | 17:05 | <pt1979@cumin2002> | START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: add nes frack firewalls - pt1979@cumin2002" | [production] | 
            
  | 17:02 | <ebernhardson@deploy1003> | helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply | [production] | 
            
  | 17:02 | <ebernhardson@deploy1003> | helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply | [production] | 
            
  | 17:01 | <pt1979@cumin2002> | START - Cookbook sre.dns.netbox | [production] | 
            
  | 16:54 | <jmm@cumin2002> | END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Manuel Merz (WMDE) out of all services on: 1552 hosts | [production] | 
            
  | 16:53 | <jmm@cumin2002> | START - Cookbook sre.idm.logout Logging Manuel Merz (WMDE) out of all services on: 1552 hosts | [production] | 
            
  | 16:52 | <jmm@cumin2002> | END (PASS) - Cookbook sre.idm.logout (exit_code=0) Logging Manuel Merz (WMDE) out of all services on: 677 hosts | [production] | 
            
  | 16:52 | <jmm@cumin2002> | START - Cookbook sre.idm.logout Logging Manuel Merz (WMDE) out of all services on: 677 hosts | [production] | 
            
  | 16:46 | <ejegg> | payments-wiki upgraded from e47e61cb to 4f3500f7 | [production] | 
            
  | 16:26 | <hnowlan@deploy1003> | Finished scap sync-world: Backport for [[gerrit:1071628|Enable async uploads on test2wiki (T356241)]] (duration: 11m 11s) | [production] | 
            
  | 16:20 | <hnowlan@deploy1003> | hnowlan: Continuing with sync | [production] | 
            
  | 16:20 | <cgoubert@cumin1002> | END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 2 hosts: mw[2428-2429].codfw.wmnet | [production] | 
            
  | 16:20 | <cgoubert@cumin1002> | START - Cookbook sre.debmonitor.remove-hosts for 2 hosts: mw[2428-2429].codfw.wmnet | [production] | 
            
  | 16:19 | <hnowlan@deploy1003> | hnowlan: Backport for [[gerrit:1071628|Enable async uploads on test2wiki (T356241)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) | [production] | 
            
  | 16:17 | <cgoubert@cumin1002> | END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2106.codfw.wmnet | [production] | 
            
  | 16:17 | <cgoubert@cumin1002> | END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2106.codfw.wmnet | [production] | 
            
  | 16:17 | <cgoubert@cumin1002> | START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2106.codfw.wmnet | [production] | 
            
  | 16:17 | <elukey@cumin1002> | END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm | [production] | 
            
  | 16:15 | <hnowlan@deploy1003> | Started scap sync-world: Backport for [[gerrit:1071628|Enable async uploads on test2wiki (T356241)]] | [production] | 
            
  | 16:13 | <elukey@cumin1002> | START - Cookbook sre.hosts.reimage for host sretest2001.codfw.wmnet with OS bookworm | [production] | 
            
  | 16:13 | <elukey@cumin1002> | END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host sretest2001.codfw.wmnet with OS bookworm | [production] | 
            
  | 16:10 | <cgoubert@cumin1002> | END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host wikikube-worker2106.codfw.wmnet with OS bullseye | [production] | 
            
  | 16:09 | <cgoubert@cumin1002> | END (PASS) - Cookbook sre.k8s.renumber-node (exit_code=0) Renumbering for host wikikube-worker2105.codfw.wmnet | [production] | 
            
  | 16:09 | <cgoubert@cumin1002> | END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2105.codfw.wmnet | [production] | 
            
  | 16:09 | <cgoubert@cumin1002> | START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2105.codfw.wmnet | [production] | 
            
  | 16:09 | <hnowlan@cumin1002> | END (PASS) - Cookbook sre.k8s.pool-depool-node (exit_code=0) pool for host wikikube-worker2095.codfw.wmnet | [production] | 
            
  | 16:09 | <hnowlan@cumin1002> | START - Cookbook sre.k8s.pool-depool-node pool for host wikikube-worker2095.codfw.wmnet | [production] | 
            
  | 16:05 | <hnowlan> | homer lsw1-b5-codfw* commit | [production] | 
            
  | 16:05 | <aqu@deploy1003> | Finished deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow (duration: 00m 10s) | [production] | 
            
  | 16:04 | <aqu@deploy1003> | Started deploy [airflow-dags/analytics_test@5315c8d]: Test Refine through Airflow | [production] | 
            
  | 16:04 | <claime> | homer lsw1-b6-codfw* commit 'T372878' | [production] | 
            
  | 16:03 | <claime> | homer cr*codfw* commit 'T372878' | [production] |