5751-5800 of 8044 results (41ms)
2019-12-25 §
16:07 <zhuyifei1999_> pkilled 5 `python pwb.py` processes belonging to `tools.kaleem-bot` on tools-sgebastion-07 [tools]
2019-12-22 §
20:13 <bd808> Enabled Puppet on tools-proxy-06.tools.eqiad.wmflabs after nginx config test (T241310) [tools]
18:52 <bd808> Disabled Puppet on tools-proxy-06.tools.eqiad.wmflabs to test nginx config change (T241310) [tools]
2019-12-20 §
22:28 <bd808> Re-enabled Puppet on tools-sgebastion-09. Reason for disable was "arturo raising systemd limits" [tools]
11:33 <arturo> reboot tools-k8s-control-3 to fix some stale NFS mount issues [tools]
2019-12-18 §
17:33 <bstorm_> updated package in aptly for toollabs-webservice to 0.53 [tools]
11:49 <arturo> introduce placeholder DNS records for toolforge.org domain. No services are provided under this domain yet for end users, this is just us testing (SSL, proxy stuff etc). This may be reverted anytime. [tools]
2019-12-17 §
20:25 <bd808> Fixed https://tools.wmflabs.org/ to redirect to https://tools.wmflabs.org/admin/ [tools]
19:20 <bstorm_> deployed the changes to the live proxy to enable the new kubernetes cluster T234037 [tools]
16:53 <bstorm_> maintain-kubeusers app deployed fully in tools for new kubernetes cluster T214513 T228499 [tools]
16:50 <bstorm_> updated the maintain-kubeusers docker image for beta and tools [tools]
04:48 <bstorm_> completed first run of maintain-kubeusers 2 in the new cluster T214513 [tools]
01:26 <bstorm_> running the first run of maintain-kubeusers 2.0 for the new cluster T214513 (more successfully this time) [tools]
01:25 <bstorm_> unset the immutable bit from 1704 tool kubeconfigs T214513 [tools]
01:05 <bstorm_> beginning the first run of the new maintain-kubeusers in gentle-mode -- but it was just killed by some files setting the immutable bit T214513 [tools]
00:45 <bstorm_> enabled encryption at rest on the new k8s cluster [tools]
2019-12-16 §
22:04 <bd808> Added 'ALLOW IPv4 25/tcp from 0.0.0.0/0' to "MTA" security group applied to tools-mail-02 [tools]
19:05 <bstorm_> deployed the maintain-kubeusers operations pod to the new cluster [tools]
2019-12-14 §
10:48 <valhallasw`cloud> re-enabling puppet on tools-sgeexec-0912, likely left-over from NFS maintenance (no reason was specified). [tools]
2019-12-13 §
18:46 <bstorm_> updated tools-k8s-control-2 and 3 to the new config as well [tools]
17:56 <bstorm_> updated tools-k8s-control-1 to the new control plane configuration [tools]
17:47 <bstorm_> edited kubeadm-config configMap object to match the new init config [tools]
17:32 <bstorm_> rebooting tools-k8s-control-2 to correct mount issue [tools]
00:44 <bstorm_> rebooting tools-static-13 [tools]
00:28 <bstorm_> rebooting the k8s master to clear NFS errors [tools]
00:15 <bstorm_> switch tools-acme-chief config to match the new authdns_servers format upstream [tools]
2019-12-12 §
23:36 <bstorm_> rebooting toolschecker after downtiming the services [tools]
22:58 <bstorm_> rebooting tools-acme-chief-01 [tools]
22:53 <bstorm_> rebooting the cron server, tools-sgecron-01 as it wasn't recovered from last night's maintenance [tools]
11:20 <arturo> rolling reboot for all grid & k8s worker nodes due to NFS staleness [tools]
09:22 <arturo> reboot tools-sgeexec-0911 to try fixing weird NFS state [tools]
08:46 <arturo> doing `run-puppet-agent` in all VMs to see state of NFS [tools]
08:34 <arturo> reboot tools-worker-1033/1034 and tools-sgebastion-08 to try to correct NFS mount issues [tools]
2019-12-11 §
18:13 <bd808> Restarted maintain-dbusers on labstore1004. Process had not logged any account creations since 2019-12-01T22:45:45. [tools]
17:24 <andrewbogott> deleted and/or truncated a bunch of logfiles on tools-worker-1031 [tools]
2019-12-10 §
13:59 <arturo> set pod replicas to 3 in the new k8s cluster (T239405) [tools]
2019-12-09 §
11:06 <andrewbogott> deleting unused security groups: catgraph, devpi, MTA, mysql, syslog, test T91619 [tools]
2019-12-04 §
13:45 <arturo> drop puppet prefix `tools-cron`, deprecated and no longer in use [tools]
2019-11-29 §
11:45 <arturo> created 3 new VMs `tools-k8s-worker-[3,4,5]` (T239403) [tools]
10:28 <arturo> re-arm keyholder in tools-acme-chief-01 (password in labs/private.git @ tools-puppetmaster-01) [tools]
10:27 <arturo> re-arm keyholder in tools-acme-chief-02 (password in labs/private.git @ tools-puppetmaster-01) [tools]
2019-11-26 §
23:25 <bstorm_> rebuilding docker images to include the new webservice 0.52 in all versions instead of just the stretch ones T236202 [tools]
22:57 <bstorm_> push upgraded webservice 0.52 to the buster and jessie repos for container rebuilds T236202 [tools]
19:55 <phamhi> drained tools-worker-1002,8,15,32 to rebalance the cluster [tools]
19:45 <phamhi> cleaned up container that was taken up 16G of disk space on tools-worker-1020 in order to re-run puppet client [tools]
14:01 <arturo> drop hiera references to `tools-test-proxy-01.tools.eqiad.wmflabs`. Such VM no longer exists [tools]
14:00 <arturo> introduce the `profile::toolforge::proxies` hiera key in the global puppet config [tools]
2019-11-25 §
10:35 <arturo> refresh puppet certs for tools-k8s-etcd-[4-6] nodes (T238655) [tools]
10:35 <arturo> add puppet cert SANs via instance hiera to tools-k8s-etcd-[4-6] nodes (T238655) [tools]
2019-11-22 §
13:32 <arturo> created security group `tools-new-k8s-full-connectivity` and add new k8s VMs to it (T238654) [tools]