7851-7900 of 8504 results (42ms)
2017-01-24 §
13:52 <yuvipanda> upgrading k8s on worker nodes to use debs + new k8s version [tools]
13:52 <yuvipanda> finished upgrading k8s + using debs [tools]
12:49 <yuvipanda> purge ancient kubectl, kube-apiserver, kube-controller-manager, kube-scheduler packages from tools-k8s-master-01, these were my old terrible packages [tools]
2017-01-23 §
19:36 <andrewbogott> temporarily shutting down tools-webgrid-lighttpd-1201 [tools]
19:35 <yuvipanda> depool tools-webgrid-lighttpd-1201 for snapshotting tests [tools]
17:13 <chasemp> reboot tools-exec-1411 as having serious transient issues [tools]
2017-01-20 §
15:58 <yuvipanda> enabling puppet across all hosts [tools]
15:36 <yuvipanda> disable puppet everywhere to cherrypick patch moving base to a profile [tools]
00:50 <bd808> sudo qdel -f 1199218 to force delete a stuck toolschecker job [tools]
2017-01-17 §
18:47 <madhuvishy> Reenabled puppet across tools [tools]
18:26 <madhuvishy> Disabling puppet across tools to test https://gerrit.wikimedia.org/r/#/c/329707/ [tools]
2017-01-11 §
22:09 <chasemp> add Reedy to admin in tool labs (approved by bryon and chase for access to investigate specific tool abuse behavior) [tools]
2017-01-10 §
19:05 <madhuvishy> Killed 3 jobs from tools.arnaub that were causing high load on tools-exec-1411 [tools]
2017-01-06 §
19:02 <bd808> Terminated deprecated instances tools-exec-121[2-6] (T154539) [tools]
2017-01-04 §
02:43 <madhuvishy> Reenabled puppet on toolschecker and removed iptables rule on labservices1001 blocking incoming connections from tools-checker-01. T152369 [tools]
2017-01-03 §
23:56 <bd808> Removed tools-exec-12[12-16] from gridengine (T154539) [tools]
23:27 <bd808> drained tools-exec-1216 (T154539) [tools]
23:26 <bd808> drained tools-exec-1215 (T154539) [tools]
23:25 <bd808> drained tools-exec-1214 (T154539) [tools]
23:25 <bd808> drained tools-exec-1213 (T154539) [tools]
23:24 <bd808> drained tools-exec-1212 (T154539) [tools]
23:11 <madhuvishy> Disabled puppet on tools-checker-01 (T152369) [tools]
21:43 <madhuvishy> Adding iptables rule to drop incoming connections from toolschecker on labservices1001 [tools]
20:51 <madhuvishy> Adding iptables rule to block outgoing connections to labservices1001 on tools-checker-01 [tools]
20:43 <madhuvishy> Silenced tools checker on icinga to test labservices1001 failure causing toolschecker to flake out T152369 [tools]
2016-12-25 §
00:28 <yuvipanda> comment out cron running 'clean' script of avicbot every minute without -once [tools]
00:28 <yuvipanda> force delete all jobs of avicbot [tools]
00:25 <yuvipanda> delete all jobs of avicbot. This is 419 jobs [tools]
00:20 <yuvipanda> kill clean.sh process of avicbot [tools]
2016-12-19 §
20:07 <valhallasw`cloud> killed gps_exif_bot2.py (tools.gpsexif), was using 50MB/s io, lagging all of tools-bastion-03 [tools]
13:06 <yuvipanda> run /usr/local/bin/deploy-master http://tools-docker-builder-03.tools.eqiad.wmflabs v1.3.3wmf1 on tools-k8s-master-01 [tools]
12:53 <yuvipanda> cleaned out pbuilder from tools-docker-builder-01 to clean up [tools]
2016-12-17 §
04:49 <yuvipanda> turned on lookupcache again for bastions [tools]
2016-12-15 §
18:52 <yuvipanda> reboot tools-exec-1204 [tools]
18:49 <yuvipanda> reboot tools-webgrid-lighttpd-12[01-05] [tools]
18:45 <yuvipanda> reboot tools-exec-gift [tools]
18:41 <yuvipanda> reboot tools-exec-1217 to 1221 [tools]
18:30 <yuvipanda> rebooted tools-exec-1212 to 1216 [tools]
14:55 <yuvipanda> reboot tools-services-01 [tools]
2016-12-14 §
18:43 <mutante> tools-bastion-03 - ran 'locale-gen ko_KR.EUC-KR' for T130532 [tools]
2016-12-13 §
20:54 <chasemp> reboot bastion-03 as unresponsive [tools]
2016-12-09 §
19:32 <godog> upgrade / restart prometheus-node-exporter [tools]
08:37 <YuviPanda> run delete-dbusers and force replica.my.cnf creation for all tools that did not have it [tools]
2016-12-08 §
18:48 <YuviPanda> restarted toolschecker on tools-checker-01 [tools]
2016-12-07 §
09:45 <YuviPanda> restart redis on tools-proxy-02 [tools]
09:32 <YuviPanda> cherry-pick https://gerrit.wikimedia.org/r/324210 and https://gerrit.wikimedia.org/r/324211 [tools]
09:29 <YuviPanda> clush -g k8s-worker -g k8s-master -g webproxy -b 'sudo puppet agent --disable "Deploying k8s change with alex"' [tools]
2016-12-06 §
00:36 <bd808> Updated toollabs-webservice to 0.31 on rest of cluster (T147350) [tools]
2016-12-05 §
23:19 <bd808> Updated toollabs-webservice to 0.31 on tools-bastion-02 (T147350) [tools]
22:55 <bd808> Updated jobutils to 1.17 on tools-mail (T147350) [tools]