|
2025-04-25
ยง
|
| 14:10 |
<bking@cumin2002> |
START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2081 |
[production] |
| 14:10 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch2081.codfw.wmnet 86.32.192.10.in-addr.arpa 6.8.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors |
[production] |
| 14:10 |
<bking@cumin2002> |
START - Cookbook sre.dns.wipe-cache cirrussearch2081.codfw.wmnet 86.32.192.10.in-addr.arpa 6.8.0.0.2.3.0.0.2.9.1.0.0.1.0.0.3.0.1.0.0.6.8.0.0.0.0.0.0.2.6.2.ip6.arpa on all recursors |
[production] |
| 14:10 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 14:10 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch2081 - bking@cumin2002" |
[production] |
| 14:10 |
<bking@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update records for host cirrussearch2081 - bking@cumin2002" |
[production] |
| 14:09 |
<fceratto@cumin1002> |
END (FAIL) - Cookbook sre.mysql.sanitize-wiki (exit_code=99) Managing sanitization for wikis nupwiki in section s5 |
[production] |
| 14:05 |
<bking@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 14:03 |
<fceratto@cumin1002> |
START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis nupwiki in section s5 |
[production] |
| 14:02 |
<andrew@cumin1002> |
START - Cookbook sre.hosts.decommission for hosts cloudlb2001-dev.codfw.wmnet |
[production] |
| 13:59 |
<Emperor> |
restart object-replicator on ms-be2089 |
[production] |
| 13:51 |
<bking@cumin2002> |
START - Cookbook sre.hosts.move-vlan for host cirrussearch2081 |
[production] |
| 13:51 |
<bking@cumin2002> |
START - Cookbook sre.hosts.reimage for host cirrussearch2081.codfw.wmnet with OS bullseye |
[production] |
| 13:51 |
<fceratto@cumin1002> |
END (FAIL) - Cookbook sre.mysql.sanitize-wiki (exit_code=99) Managing sanitization for wikis nupwiki in section s5 |
[production] |
| 13:51 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cirrussearch2081.codfw.wmnet on all recursors |
[production] |
| 13:51 |
<bking@cumin2002> |
START - Cookbook sre.dns.wipe-cache cirrussearch2081.codfw.wmnet on all recursors |
[production] |
| 13:49 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.hosts.rename (exit_code=0) from elastic2081 to cirrussearch2081 |
[production] |
| 13:48 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cirrussearch2081 |
[production] |
| 13:48 |
<bking@cumin2002> |
START - Cookbook sre.network.configure-switch-interfaces for host cirrussearch2081 |
[production] |
| 13:48 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.dns.netbox (exit_code=0) |
[production] |
| 13:47 |
<bking@cumin2002> |
END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming elastic2081 to cirrussearch2081 - bking@cumin2002" |
[production] |
| 13:47 |
<bking@cumin2002> |
START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Renaming elastic2081 to cirrussearch2081 - bking@cumin2002" |
[production] |
| 13:46 |
<fceratto@cumin1002> |
START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis nupwiki in section s5 |
[production] |
| 13:45 |
<fceratto@cumin1002> |
END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis nupwiki in section s5 |
[production] |
| 13:43 |
<taavi@cumin2002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2004-dev.codfw.wmnet |
[production] |
| 13:43 |
<fceratto@cumin1002> |
START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis nupwiki in section s5 |
[production] |
| 13:36 |
<fceratto@cumin1002> |
END (FAIL) - Cookbook sre.mysql.sanitize-wiki (exit_code=99) Managing sanitization for wikis nupwiki in section s5 |
[production] |
| 13:34 |
<taavi@cumin2002> |
START - Cookbook sre.hosts.reboot-single for host cloudlb2004-dev.codfw.wmnet |
[production] |
| 13:33 |
<taavi> |
add cloudlb2004-dev bgp session to cloudsw1-b1-codfw T377126 |
[production] |
| 13:33 |
<bking@cumin2002> |
START - Cookbook sre.dns.netbox |
[production] |
| 13:32 |
<bking@cumin2002> |
START - Cookbook sre.hosts.rename from elastic2081 to cirrussearch2081 |
[production] |
| 13:31 |
<fceratto@cumin1002> |
START - Cookbook sre.mysql.sanitize-wiki Managing sanitization for wikis nupwiki in section s5 |
[production] |
| 13:29 |
<fceratto@cumin1002> |
END (PASS) - Cookbook sre.mysql.sanitize-wiki (exit_code=0) Checking sanitization for wikis nupwiki in section s5 |
[production] |
| 13:26 |
<fceratto@cumin1002> |
START - Cookbook sre.mysql.sanitize-wiki Checking sanitization for wikis nupwiki in section s5 |
[production] |
| 13:08 |
<jmm@cumin2002> |
END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host sretest1001.eqiad.wmnet with OS trixie |
[production] |
| 12:58 |
<vgutierrez> |
restarting grafana-server.service @ grafana1002.eqiad.wmnet |
[production] |
| 12:55 |
<taavi@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218 |
[admin] |
| 12:54 |
<taavi@cloudcumin1001> |
START - Cookbook wmcs.openstack.tofu running tofu plan for https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/218 |
[admin] |
| 12:30 |
<aborrero@cloudcumin1001> |
END (PASS) - Cookbook wmcs.openstack.tofu (exit_code=0) running tofu plan for main branch |
[admin] |
| 12:30 |
<aborrero@cloudcumin1001> |
START - Cookbook wmcs.openstack.tofu running tofu plan for main branch |
[admin] |
| 11:53 |
<jmm@cumin2002> |
END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage |
[production] |
| 11:49 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.downtime for 2:00:00 on sretest1001.eqiad.wmnet with reason: host reimage |
[production] |
| 11:31 |
<jmm@cumin2002> |
START - Cookbook sre.hosts.reimage for host sretest1001.eqiad.wmnet with OS trixie |
[production] |
| 09:31 |
<moritzm> |
restarting puppetserver on puppetserver1002 (apparently needs a restart which per timing seems related to https://gerrit.wikimedia.org/r/c/operations/puppet/+/1138904) |
[production] |
| 09:16 |
<vgutierrez> |
restarting puppetserver on puppetserver1003 |
[production] |
| 09:11 |
<taavi> |
removed cloudlb2001-dev bgp session from cloudsw1-b1-codfw T377126 |
[production] |
| 08:36 |
<joal> |
rerun airflow mediawiki_history_metrics_monthly |
[analytics] |
| 08:36 |
<joal> |
rerun airflow mediawiki_history_check_denormalize |
[analytics] |
| 08:24 |
<marostegui@cumin1002> |
dbctl commit (dc=all): 'Promote es2032 to es1 master T391921', diff saved to https://phabricator.wikimedia.org/P75463 and previous config saved to /var/cache/conftool/dbconfig/20250425-082420-marostegui.json |
[production] |
| 07:50 |
<taavi@cumin1002> |
END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudlb2002-dev.codfw.wmnet |
[production] |