8251-8300 of 9335 results (37ms)
2018-01-17 §
17:55 <arturo> aborrero@tools-clushmaster-01:~$ clush -w @all 'sudo report-pending-upgrades -v' | tee pending-upgrades-report.txt [tools]
15:15 <andrewbogott> running purge-old-kernels on all Trusty exec nodes [tools]
15:15 <andrewbogott> repooling exec-manage tools-exec-1430. [tools]
15:04 <andrewbogott> depooling exec-manage tools-exec-1430. Experimenting with purge-old-kernels [tools]
14:09 <arturo> T181647 aborrero@tools-clushmaster-01:~$ clush -w @all 'sudo puppet agent --test' [tools]
2018-01-16 §
22:01 <chasemp> qstat -explain E -xml | grep 'name' | sed 's/<name>//' | sed 's/<\/name>//' | xargs qmod -cq [tools]
21:54 <chasemp> tools-exec-1436:~$ /sbin/reboot [tools]
21:24 <andrewbogott> repooled tools-exec-1420 and tools-webgrid-lighttpd-1417 [tools]
21:14 <andrewbogott> depooling tools-exec-1420 and tools-webgrid-lighttpd-1417 [tools]
20:58 <andrewbogott> depooling tools-exec-1412, 1415, 1417, tools-webgrid-lighttpd-1415, 1416, 1422, 1426 [tools]
20:56 <andrewbogott> repooling tools-exec-1416, 1418, 1424, tools-webgrid-lighttpd-1404, tools-webgrid-lighttpd-1410 [tools]
20:46 <andrewbogott> depooling tools-exec-1416, 1418, 1424, tools-webgrid-lighttpd-1404, tools-webgrid-lighttpd-1410 [tools]
20:46 <andrewbogott> repooled tools-exec-1406, 1421, 1436, 1437, tools-webgrid-generic-1404, tools-webgrid-lighttpd-1409, tools-webgrid-lighttpd-1411, tools-webgrid-lighttpd-1418, tools-webgrid-lighttpd-1425 [tools]
20:33 <andrewbogott> depooling tools-exec-1406, 1421, 1436, 1437, tools-webgrid-generic-1404, tools-webgrid-lighttpd-1409, tools-webgrid-lighttpd-1411, tools-webgrid-lighttpd-1418, tools-webgrid-lighttpd-1425 [tools]
20:20 <andrewbogott> depooling tools-webgrid-lighttpd-1412 and tools-exec-1423 for host reboot [tools]
20:19 <andrewbogott> repooled tools-exec-1409, 1410, 1414, 1419, 1427, 1428 tools-webgrid-generic-1401, tools-webgrid-lighttpd-1406 [tools]
20:02 <andrewbogott> depooling tools-exec-1409, 1410, 1414, 1419, 1427, 1428 tools-webgrid-generic-1401, tools-webgrid-lighttpd-1406 [tools]
20:00 <andrewbogott> depooled and repooled tools-webgrid-lighttpd-1427 tools-webgrid-lighttpd-1428 tools-exec-1413 tools-exec-1442 for host reboot [tools]
18:50 <andrewbogott> switched active proxy back to tools-proxy-02 [tools]
18:50 <andrewbogott> repooling tools-exec-1422 and tools-webgrid-lighttpd-1413 [tools]
18:34 <andrewbogott> moving proxy from tools-proxy-02 to tools-proxy-01 [tools]
18:31 <andrewbogott> depooling tools-exec-1422 and tools-webgrid-lighttpd-1413 for host reboot [tools]
18:26 <andrewbogott> repooling tools-exec-1404 and 1434 for host reboot [tools]
18:06 <andrewbogott> depooling tools-exec-1404 and 1434 for host reboot [tools]
18:04 <andrewbogott> repooling tools-exec-1402, 1426, 1429, 1433, tools-webgrid-lighttpd-1408, 1414, 1424 [tools]
17:48 <andrewbogott> depooling tools-exec-1402, 1426, 1429, 1433, tools-webgrid-lighttpd-1408, 1414, 1424 [tools]
17:28 <andrewbogott> disabling tools-webgrid-generic-1402, tools-webgrid-lighttpd-1403, tools-exec-1403 for host reboot [tools]
17:26 <andrewbogott> repooling tools-exec-1405, 1425, tools-webgrid-generic-1403, tools-webgrid-lighttpd-1401, 1405 after host reboot [tools]
17:08 <andrewbogott> depooling tools-exec-1405, 1425, tools-webgrid-generic-1403, tools-webgrid-lighttpd-1401, 1405 for host reboot [tools]
16:19 <andrewbogott> repooling tools-exec-1401, 1407, 1408, 1430, 1431, 1432, 1435, 1438, 1439, 1441, tools-webgrid-lighttpd-1402, 1407 after host reboot [tools]
15:52 <andrewbogott> depooling tools-exec-1401, 1407, 1408, 1430, 1431, 1432, 1435, 1438, 1439, 1441, tools-webgrid-lighttpd-1402, 1407 for host reboot [tools]
13:35 <chasemp> tools-mail almouked@ltnet.net 719 pending messages cleared [tools]
2018-01-11 §
20:33 <andrewbogott> repooling tools-exec-1411, tools-exec-1440, tools-webgrid-lighttpd-1419, tools-webgrid-lighttpd-1420, tools-webgrid-lighttpd-1421 [tools]
20:33 <andrewbogott> uncordoning tools-worker-1012 and tools-worker-1017 [tools]
20:06 <andrewbogott> cordoning tools-worker-1012 and tools-worker-1017 [tools]
20:02 <andrewbogott> depooling tools-exec-1411, tools-exec-1440, tools-webgrid-lighttpd-1419, tools-webgrid-lighttpd-1420, tools-webgrid-lighttpd-1421 [tools]
19:00 <chasemp> reboot tools-worker-1015 [tools]
15:08 <chasemp> reboot tools-exec-1405 [tools]
15:06 <chasemp> reboot tools-exec-1404 [tools]
15:06 <chasemp> reboot tools-exec-1403 [tools]
15:02 <chasemp> reboot tools-exec-1402 [tools]
14:57 <chasemp> reboot tools-exec-1401 again... [tools]
14:53 <chasemp> reboot tools-exec-1401 [tools]
14:46 <chasemp> install metltdown kernel and reboot workers 1011-1016 as jessie pilot [tools]
2018-01-10 §
15:14 <chasemp> tools-clushmaster-01:~$ clush -f 1 -w @k8s-worker "sudo puppet agent --enable && sudo puppet agent --test" [tools]
15:03 <chasemp> tools-k8s-master-01:~# for n in `kubectl get nodes | awk '{print $1}' | grep -v -e tools-worker-1001 -e tools-worker-1016 -e tools-worker-1016`; do kubectl cordon $n; done [tools]
14:41 <chasemp> tools-clushmaster-01:~$ clush -w @k8s-worker "sudo puppet agent --disable 'chase rollout'" [tools]
14:01 <chasemp> tools-k8s-master-01:~# kubectl uncordon tools-worker-1001.tools.eqiad.wmflabs [tools]
13:57 <arturo> T184604 cleaned stalled log files that prevented logrotate from working. Triggered a couple of logrorate runs by hand in tools-worker-1020.tools.eqiad.wmflabs [tools]
13:46 <arturo> T184604 aborrero@tools-k8s-master-01:~$ sudo kubectl uncordon tools-worker-1020.tools.eqiad.wmflabs [tools]