801-850 of 1984 results (15ms)
2018-02-23 §
18:53 <arturo> disable puppet in tools-proxy-01 for apt preferences testing [tools]
13:52 <arturo> deploying https://gerrit.wikimedia.org/r/#/c/413725/ across the fleet [tools]
13:04 <arturo> install apt-rdepends in tools-paws-master-01 which triggered some python libs to be upgraded [tools]
2018-02-22 §
16:31 <bstorm_> Enabled puppet on tools-static-12 as the test server [tools]
2018-02-21 §
19:02 <bstorm_> disabled puppet on tools-static-* pending change 413197 [tools]
18:15 <arturo> puppet should be fine across the fleet [tools]
17:24 <arturo> another try: merged https://gerrit.wikimedia.org/r/#/c/413202/ [tools]
17:02 <arturo> revert last change https://gerrit.wikimedia.org/r/#/c/413198/ [tools]
16:59 <arturo> puppet is broken across the cluster due to last change [tools]
16:57 <arturo> deploying https://gerrit.wikimedia.org/r/#/c/410177/ [tools]
16:26 <bd808> Rebooting tools-docker-registry-01, NFS mounts are in a bad state [tools]
11:43 <arturo> package upgrades in tools-webgrid-lightttpd-1401 [tools]
11:35 <arturo> package upgrades in tools-package-builder-01 tools-prometheus-01 tools-static-10 and tools-redis-1001 [tools]
11:22 <arturo> package upgrades in tools-mail, tools-grid-master, tool-logs-02 [tools]
10:51 <arturo> package upgrades in tools-checker-01 tools-clushmaster-01 and tools-docker-builder-05 [tools]
09:18 <chicocvenancio> killed io intensive tool job in bastion [tools]
03:32 <zhuyifei1999_> removed /data/project/.elasticsearch.ini, owned by root and mode 644, leaks the creds of /data/project/strephit/.elasticsearch.ini Might need to cycle it as well... [tools]
2018-02-20 §
12:42 <arturo> upgrading tools-flannel-etcd-01 [tools]
12:42 <arturo> upgrading tools-k8s-etcd-01 [tools]
2018-02-19 §
19:13 <arturo> upgrade all packages of tools-services-01 [tools]
19:02 <arturo> move tools-services-01 from puppet3 to puppet4 (manual package upgrade). No issues detected. [tools]
18:23 <arturo> upgrade packages of tools-cron-01 from all channels (trusty-wikimedia, trusty-updates and trusty-tools) [tools]
12:54 <arturo> puppet run with clush to ensure puppet is back to normal after being broken due to duplicated python3 declaration [tools]
2018-02-16 §
18:21 <arturo> upgrading tools-proxy-01 and tools-paws-master-01, same as others [tools]
17:36 <arturo> upgrading oldstable, jessie-backports, jessie-wikimedia packages in tools-k8s-master-01 (excluding linux*, libpam*, nslcd) [tools]
13:00 <arturo> upgrades in tools-exec-14[01-10].eqiad.wmflabs were fine [tools]
12:42 <arturo> aborrero@tools-clushmaster-01:~$ clush -q -w @exec-upgrade-canarys 'DEBIAN_FRONTEND=noninteractive sudo apt-upgrade -u upgrade trusty-updates -y' [tools]
11:58 <arturo> aborrero@tools-elastic-01:~$ sudo apt-upgrade -u -f exclude upgrade jessie-wikimedia -y [tools]
11:57 <arturo> aborrero@tools-elastic-01:~$ sudo apt-upgrade -u -f exclude upgrade jessie-backports -y [tools]
11:53 <arturo> (10 exec canary nodes) aborrero@tools-clushmaster-01:~$ clush -q -w @exec-upgrade-canarys 'sudo apt-upgrade -u upgrade trusty-wikimedia -y' [tools]
11:41 <arturo> aborrero@tools-elastic-01:~$ sudo apt-upgrade -u -f exclude upgrade oldstable -y [tools]
2018-02-15 §
13:54 <arturo> cleanup ferm (deinstall) in tools-services-01 for T187435 [tools]
13:28 <arturo> aborrero@tools-bastion-02:~$ sudo apt-upgrade -u upgrade trusty-tools [tools]
13:16 <arturo> aborrero@tools-bastion-02:~$ sudo apt-upgrade -u upgrade trusty-updates -y [tools]
13:13 <arturo> aborrero@tools-bastion-02:~$ sudo apt-upgrade -u upgrade trusty-wikimedia [tools]
13:06 <arturo> aborrero@tools-webgrid-generic-1401:~$ sudo apt-upgrade -u upgrade trusty-tools [tools]
12:57 <arturo> aborrero@tools-webgrid-generic-1401:~$ sudo apt-upgrade -u upgrade trusty-updates [tools]
12:51 <arturo> aborrero@tools-webgrid-generic-1401:~$ sudo apt-upgrade -u upgrade trusty-wikimedia [tools]
2018-02-14 §
13:09 <arturo> the reboot was OK, the server seems working and kubectl sees all the pods running in the deployment (T187315) [tools]
13:04 <arturo> reboot tools-paws-master-01 for T187315 [tools]
2018-02-11 §
01:28 <zhuyifei1999_> `# find /home/ -maxdepth 1 -perm -o+w \! -uid 0 -exec chmod -v o-w {} \;` Affected: only /home/tr8dr, mode 0777 -> 0775 [tools]
01:21 <zhuyifei1999_> `# find /data/project/ -maxdepth 1 -perm -o+w \! -uid 0 -exec chmod -v o-w {} \;` Affected tools: wikisource-tweets, gsociftttdev, dow, ifttt-testing, elobot. All mode 2777 -> 2775 [tools]
2018-02-09 §
10:35 <arturo> deploy https://gerrit.wikimedia.org/r/#/c/409226/ T179343 T182562 T186846 [tools]
06:15 <bd808> Killed orphan processes owned by iabot, dupdet, and wsexport scattered across the webgrid nodes [tools]
05:07 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1426 [tools]
05:06 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1411 [tools]
05:05 <bd808> Killed 1 orphan php-fcgi process from jembot that were running on tools-webgrid-lighttpd-1409 [tools]
05:02 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1421 and pegging the cpu there [tools]
04:56 <bd808> Rescheduled 30 of the 60 tools running on tools-webgrid-lighttpd-1421 (T186830) [tools]
04:39 <bd808> Killed 4 orphan php-fcgi processes from jembot that were running on tools-webgrid-lighttpd-1417 and pegging the cpu there [tools]