2251-2300 of 10000 results (121ms)
2024-10-28 §
07:57 <kartik@deploy2002> kartik: Continuing with sync [production]
07:56 <kartik@deploy2002> kartik: Backport for [[gerrit:1083292|Disable MT in Content Translation on Lithuanian Wikipedia (T364073)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) [production]
07:45 <kartik@deploy2002> Started scap sync-world: Backport for [[gerrit:1083292|Disable MT in Content Translation on Lithuanian Wikipedia (T364073)]] [production]
07:14 <arnaudb@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[1169,1234].eqiad.wmnet with reason: maintenance [production]
07:14 <arnaudb@cumin1002> START - Cookbook sre.hosts.downtime for 4:00:00 on db[1169,1234].eqiad.wmnet with reason: maintenance [production]
06:07 <taavi@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: replication broken T378320 [production]
06:06 <taavi@cumin1002> START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: replication broken T378320 [production]
06:03 <taavi@cumin1002> dbctl commit (dc=all): 'depool db1169', diff saved to https://phabricator.wikimedia.org/P70590 and previous config saved to /var/cache/conftool/dbconfig/20241028-060327-taavi.json [production]
2024-10-27 §
13:41 <Dreamy_Jazz> Starting MediaModeration scanning on group1 wikis [production]
13:37 <Dreamy_Jazz> Starting MediaModeration scanning on group2 wikis [production]
2024-10-26 §
16:29 <mvernon@cumin1002> dbctl commit (dc=all): 'Depool db1234', diff saved to https://phabricator.wikimedia.org/P70589 and previous config saved to /var/cache/conftool/dbconfig/20241026-162946-mvernon.json [production]
16:29 <mvernon@cumin1002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db1234.eqiad.wmnet with reason: spontaneous reboot, depooling 'til Monday [production]
16:28 <mvernon@cumin1002> START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db1234.eqiad.wmnet with reason: spontaneous reboot, depooling 'til Monday [production]
02:03 <tzatziki> removing 9 files for legal compliance [production]
2024-10-25 §
18:29 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup2012.codfw.wmnet with OS bookworm [production]
18:28 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
18:18 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jhancock@cumin2002" [production]
17:54 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup2012.codfw.wmnet with reason: host reimage [production]
17:50 <jhancock@cumin2002> START - Cookbook sre.hosts.downtime for 2:00:00 on backup2012.codfw.wmnet with reason: host reimage [production]
16:42 <jhancock@cumin2002> START - Cookbook sre.hosts.reimage for host backup2012.codfw.wmnet with OS bookworm [production]
16:32 <jhancock@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host backup2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
16:28 <JustHannah> T378170 Ran mwscript-k8s extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=trwiki --logwiki=metawiki 'Peter.kerepesi' 'Peakbagger77' @ 11:57:19 UTC [production]
15:59 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host backup2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:57 <jhancock@cumin2002> END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host backup2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:56 <jhancock@cumin2002> START - Cookbook sre.hosts.provision for host backup2012.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
15:53 <jhancock@cumin2002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host backup2012 [production]
15:52 <jhancock@cumin2002> START - Cookbook sre.network.configure-switch-interfaces for host backup2012 [production]
15:52 <jhancock@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
15:52 <jhancock@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding backup2012 to codfw - jhancock@cumin2002" [production]
15:51 <jhancock@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: adding backup2012 to codfw - jhancock@cumin2002" [production]
15:47 <jhancock@cumin2002> START - Cookbook sre.dns.netbox [production]
15:07 <dancy@deploy2002> Installation of scap version "4.118.0" completed for 209 hosts [production]
15:03 <dancy@deploy2002> Installing scap version "4.118.0" for 209 hosts [production]
14:31 <herron> alert1002: manually killed stunnel4 process to clear puppet failure T375143 [production]
14:02 <sukhe> running authdns-update for CR 1082548 [production]
10:31 <arnaudb@cumin1002> dbctl commit (dc=all): 'maintenance', diff saved to https://phabricator.wikimedia.org/P70588 and previous config saved to /var/cache/conftool/dbconfig/20241025-103157-arnaudb.json [production]
10:21 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
10:18 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host sretest2001.mgmt.codfw.wmnet with chassis set policy GRACEFUL_RESTART [production]
10:17 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host sretest1001.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
10:16 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host sretest1001.mgmt.eqiad.wmnet with chassis set policy GRACEFUL_RESTART and with Dell SCP reboot policy GRACEFUL [production]
10:15 <elukey@cumin2002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ms-be2081.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
10:12 <elukey@cumin2002> START - Cookbook sre.hosts.provision for host ms-be2081.mgmt.codfw.wmnet with chassis set policy FORCE_RESTART [production]
09:56 <jmm@cumin2002> END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2014.codfw.wmnet [production]
09:17 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2012.codfw.wmnet [production]
09:17 <jmm@cumin2002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
09:17 <jmm@cumin2002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2012.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
09:17 <jmm@cumin2002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: ganeti2012.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002" [production]
09:12 <jmm@cumin2002> START - Cookbook sre.dns.netbox [production]
09:06 <jmm@cumin2002> START - Cookbook sre.hosts.decommission for hosts ganeti2012.codfw.wmnet [production]
09:05 <jmm@cumin2002> END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts ganeti2011.codfw.wmnet [production]