5151-5200 of 10000 results (129ms)
2024-12-10 ยง
17:00 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-video: apply [production]
16:59 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-timeline: apply [production]
16:59 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-timeline: apply [production]
16:59 <elukey@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-lab1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
16:59 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
16:58 <elukey@cumin1002> START - Cookbook sre.hosts.provision for host ml-lab1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
16:58 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]
16:58 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-media: apply [production]
16:57 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-media: apply [production]
16:57 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox-constraints: apply [production]
16:57 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox-constraints: apply [production]
16:56 <swfrench@deploy2002> helmfile [eqiad] DONE helmfile.d/services/shellbox: apply [production]
16:56 <swfrench@deploy2002> helmfile [eqiad] START helmfile.d/services/shellbox: apply [production]
16:51 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-video: apply [production]
16:51 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-video: apply [production]
16:50 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-timeline: apply [production]
16:50 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-timeline: apply [production]
16:50 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-syntaxhighlight: apply [production]
16:49 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-syntaxhighlight: apply [production]
16:49 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-media: apply [production]
16:49 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-media: apply [production]
16:49 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox-constraints: apply [production]
16:48 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox-constraints: apply [production]
16:48 <swfrench@deploy2002> helmfile [staging] DONE helmfile.d/services/shellbox: apply [production]
16:47 <swfrench@deploy2002> helmfile [staging] START helmfile.d/services/shellbox: apply [production]
16:38 <denisse@deploy2002> Finished deploy [librenms/librenms@f049593]: Upgrade LibreNMS to 24.10.0 - T381785 (duration: 00m 13s) [production]
16:38 <denisse@deploy2002> Started deploy [librenms/librenms@f049593]: Upgrade LibreNMS to 24.10.0 - T381785 [production]
16:26 <klausman@cumin1002> START - Cookbook sre.hosts.reimage for host ml-lab1002.eqiad.wmnet with OS bookworm [production]
16:25 <klausman@cumin1002> END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ml-lab1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
16:20 <klausman@cumin1002> START - Cookbook sre.hosts.provision for host ml-lab1002.mgmt.eqiad.wmnet with chassis set policy FORCE_RESTART [production]
16:15 <klausman@cumin1002> END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host ml-lab1002 [production]
16:15 <klausman@cumin1002> START - Cookbook sre.network.configure-switch-interfaces for host ml-lab1002 [production]
16:13 <hnowlan@deploy1003> helmfile [codfw] DONE helmfile.d/services/mw-videoscaler: apply [production]
16:13 <hnowlan@deploy1003> helmfile [codfw] START helmfile.d/services/mw-videoscaler: apply [production]
16:13 <moritzm> installing postgresql-15 security updates [production]
16:12 <klausman@cumin1002> END (PASS) - Cookbook sre.dns.netbox (exit_code=0) [production]
16:12 <klausman@cumin1002> END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update DNS for newly-provisioned ml-lab1002 - klausman@cumin1002" [production]
16:12 <klausman@cumin1002> START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Update DNS for newly-provisioned ml-lab1002 - klausman@cumin1002" [production]
16:09 <elukey@deploy2002> helmfile [codfw] DONE helmfile.d/admin 'sync'. [production]
16:09 <dzahn@cumin2002> END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:10:00 on phab1004.eqiad.wmnet with reason: nftables [production]
16:09 <elukey@deploy2002> helmfile [codfw] START helmfile.d/admin 'sync'. [production]
16:09 <mutante> phabricator production host needs a maintenance reboot - expect short downtime [production]
16:09 <dzahn@cumin2002> START - Cookbook sre.hosts.downtime for 0:10:00 on phab1004.eqiad.wmnet with reason: nftables [production]
16:09 <elukey@deploy2002> helmfile [eqiad] DONE helmfile.d/admin 'sync'. [production]
16:08 <elukey@deploy2002> helmfile [eqiad] START helmfile.d/admin 'sync'. [production]
16:08 <klausman@cumin1002> START - Cookbook sre.dns.netbox [production]
16:07 <moritzm> manually clean out ganeti1009 from puppetdb, decom cookbook got interrupted T381652 [production]
16:06 <jmm@cumin2002> END (PASS) - Cookbook sre.debmonitor.remove-hosts (exit_code=0) for 1 hosts: ganeti1009.eqiad.wmnet [production]
16:06 <jmm@cumin2002> START - Cookbook sre.debmonitor.remove-hosts for 1 hosts: ganeti1009.eqiad.wmnet [production]
16:03 <marostegui@cumin1002> dbctl commit (dc=all): 'db1169 (re)pooling @ 100%: Pooling in production', diff saved to https://phabricator.wikimedia.org/P71685 and previous config saved to /var/cache/conftool/dbconfig/20241210-160322-root.json [production]