|
2025-04-30
§
|
| 14:44 |
<mszabo@deploy1003> |
mszabo: Continuing with sync |
[production] |
| 14:42 |
<mszabo@deploy1003> |
mszabo: Backport for [[gerrit:1140191|popup: Fix target user name for expired temporary account links (T393002)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug) |
[production] |
| 14:42 |
<sukhe@dns1004> |
START - running authdns-update |
[production] |
| 14:41 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P75713 and previous config saved to /var/cache/conftool/dbconfig/20250430-144144-fceratto.json |
[production] |
| 14:40 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2218.codfw.wmnet with reason: Maintenance |
[production] |
| 14:40 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1236.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:36 |
<mszabo@deploy1003> |
Started scap sync-world: Backport for [[gerrit:1140191|popup: Fix target user name for expired temporary account links (T393002)]] |
[production] |
| 14:34 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2003.codfw.wmnet |
[production] |
| 14:29 |
<moritzm> |
installing ruby2.7 security updates |
[production] |
| 14:28 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be2003.codfw.wmnet |
[production] |
| 14:27 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2002.codfw.wmnet |
[production] |
| 14:26 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1185 (T392806)', diff saved to https://phabricator.wikimedia.org/P75712 and previous config saved to /var/cache/conftool/dbconfig/20250430-142636-fceratto.json |
[production] |
| 14:21 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be2002.codfw.wmnet |
[production] |
| 14:20 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be2001.codfw.wmnet |
[production] |
| 14:18 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Depooling db1185 (T392806)', diff saved to https://phabricator.wikimedia.org/P75711 and previous config saved to /var/cache/conftool/dbconfig/20250430-141845-fceratto.json |
[production] |
| 14:18 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1185.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:18 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1183 (T392806)', diff saved to https://phabricator.wikimedia.org/P75710 and previous config saved to /var/cache/conftool/dbconfig/20250430-141819-fceratto.json |
[production] |
| 14:14 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be2001.codfw.wmnet |
[production] |
| 14:14 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-cluster (exit_code=0) |
[production] |
| 14:12 |
<hnowlan@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply |
[production] |
| 14:12 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki1001.eqiad.wmnet |
[production] |
| 14:12 |
<hnowlan@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-cron: apply |
[production] |
| 14:11 |
<moritzm> |
failover Ganeti master in codfw to ganeti2021 |
[production] |
| 14:09 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host rpki1001.eqiad.wmnet |
[production] |
| 14:06 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host rpki2003.codfw.wmnet |
[production] |
| 14:05 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.vps.add_user_to_project (exit_code=0) for user 'andreadenina' in role 'reader' |
[bastion] |
| 14:04 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2209.codfw.wmnet with reason: Maintenance |
[production] |
| 14:04 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.vps.add_user_to_project for user 'andreadenina' in role 'reader' |
[bastion] |
| 14:03 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P75709 and previous config saved to /var/cache/conftool/dbconfig/20250430-140312-fceratto.json |
[production] |
| 14:02 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host rpki2003.codfw.wmnet |
[production] |
| 14:01 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1223.eqiad.wmnet with reason: Maintenance |
[production] |
| 14:00 |
<moritzm> |
installing libcap2 security updates |
[production] |
| 13:54 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-cluster |
[production] |
| 13:52 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2044.codfw.wmnet |
[production] |
| 13:52 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2044.codfw.wmnet |
[production] |
| 13:52 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be1003.eqiad.wmnet |
[production] |
| 13:50 |
<Lucas_WMDE> |
UTC afternoon backport+config window done |
[production] |
| 13:48 |
<fceratto@cumin1002> |
dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P75708 and previous config saved to /var/cache/conftool/dbconfig/20250430-134805-fceratto.json |
[production] |
| 13:47 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host ganeti2044.codfw.wmnet |
[production] |
| 13:46 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be1003.eqiad.wmnet |
[production] |
| 13:44 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be1002.eqiad.wmnet |
[production] |
| 13:43 |
<hnowlan@deploy1003> |
helmfile [eqiad] DONE helmfile.d/services/mw-cron: apply |
[production] |
| 13:43 |
<hnowlan@deploy1003> |
helmfile [eqiad] START helmfile.d/services/mw-cron: apply |
[production] |
| 13:38 |
<mvernon@cumin1002> |
START - Cookbook sre.hosts.reboot-single for host moss-be1002.eqiad.wmnet |
[production] |
| 13:36 |
<mvernon@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host moss-be1001.eqiad.wmnet |
[production] |
| 13:36 |
<jmm@cumin2002> |
START - Cookbook sre.ganeti.drain-node for draining ganeti node ganeti2044.codfw.wmnet |
[production] |
| 13:35 |
<urandom> |
invoking `nodetool garbagecollect` on sessionstore1004 — T392989, T390514 |
[production] |
| 13:35 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.ganeti.drain-node (exit_code=0) for draining ganeti node ganeti2043.codfw.wmnet |
[production] |
| 13:35 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host ganeti2043.codfw.wmnet |
[production] |
| 13:33 |
<fceratto@cumin1002> |
DONE (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2229.codfw.wmnet with reason: Maintenance |
[production] |