1101-1150 of 1438 results (15ms)
2016-06-08 §
20:31 <yuvipanda> start tools-bastion-03 was stuck in 'stopped' state [tools]
20:31 <yuvipanda> reboot tools-bastion-03 [tools]
2016-05-31 §
17:35 <valhallasw`cloud> re-enabled queues on tools-exec-1407, tools-exec-1216, tools-exec-1219 [tools]
13:13 <chasemp> reboot of tools-exec-1203 see T136495 all jobs seem gone now [tools]
2016-05-30 §
13:06 <valhallasw`cloud> rebooting tools-exec-1221 [tools]
11:53 <godog> cherry-pick https://gerrit.wikimedia.org/r/#/c/280652 https://gerrit.wikimedia.org/r/#/c/290479 https://gerrit.wikimedia.org/r/#/c/291710/ on tools-puppetmaster-01 [tools]
2016-05-29 §
18:58 <YuviPanda> deleted tools-k8s-bastion-01 for T136496 [tools]
14:29 <valhallasw`cloud> chowned /data/project/xtools-mab-dev to root and back to stop rogue process that was writing to the directory. I'm still not sure where that process was running, but at least this seems to have solved the issue [tools]
2016-05-28 §
21:51 <valhallasw`cloud> rebooted tools-webgrid-lighttpd-1408, tools-pastion-01, tools-exec-1205 [tools]
21:21 <valhallasw`cloud> rebooting tools-exec-1204 (T136495) [tools]
2016-05-27 §
14:45 <YuviPanda> start moving tools-bastion-03 to use tools-puppetmaster-01 as puppetmaster [tools]
2016-05-25 §
20:15 <YuviPanda> deleted tools-bastion-mtemp per chasemp [tools]
19:43 <YuviPanda> delete devpi instance, not currently in use [tools]
19:39 <YuviPanda> run sudo dpkg --configure -a on tools-worker-1007 to get it unstuck [tools]
19:19 <YuviPanda> deleted tools-docker-builder-01 and -02, hosed hosts that are unused [tools]
17:18 <YuviPanda> fixed hhvm upgrade on tools-cron-01 [tools]
07:19 <YuviPanda> hard reboot tools-services-01, was completely stuck on /public/dumps [tools]
06:06 <bd808> Restarting all webservice jobs [tools]
05:33 <andrewbogott> rebooting tools-proxy-02 [tools]
2016-05-23 §
19:36 <YuviPanda> switched tools-checker to tools-checker-03 [tools]
13:28 <chasemp> 'apt-get install hhvm -y --force-yes' across trusty hosts to handle hhvm downgrade [tools]
2016-05-20 §
23:39 <bd808> Forced puppet run on bastion-02 & bastion-05 to apply fix for T135861 [tools]
19:44 <chasemp> tools-exec-1406 having issues rebooting [tools]
2016-05-19 §
21:07 <bd808> deployed jobutils 1.13 on bastions; now with '-l release=...' validation! [tools]
15:43 <YuviPanda> rebooting all tools worker instances [tools]
13:12 <chasemp> reboot tools-exec-1220 stuck in state of unresponsivenss [tools]
2016-05-13 §
00:40 <YuviPanda> cleared all queues that were in error state [tools]
2016-05-12 §
22:59 <YuviPanda> restart tools-worker-1004 to attempt bringing it back up [tools]
22:59 <YuviPanda> deploy k8s 1.2.4wmf1 on all proxy nodes [tools]
22:58 <YuviPanda> deploy k8s on all worker nodes [tools]
22:46 <YuviPanda> deploy k8s master for 1.2.4wmf1 [tools]
2016-05-10 §
04:25 <bd808> Added role::package::builder to tools-services-01 [tools]
2016-05-09 §
04:33 <YuviPanda> reboot tools-worker-1004, lots of ksoftirqd stuckness despite no actual containers running [tools]
2016-05-08 §
07:06 <YuviPanda> restarted admin tool [tools]
2016-05-05 §
13:11 <godog> cherry-pick https://gerrit.wikimedia.org/r/#/c/280652/ on puppetmaster [tools]
2016-04-28 §
04:15 <YuviPanda> delete half of the trusty webservice jobs [tools]
04:00 <YuviPanda> deleted all precise webservice jobs, waiting for webservicemonitor to bring them back up [tools]
2016-04-24 §
12:22 <YuviPanda> force deleted job 5435259 from pbbot per PeterBowman [tools]
2016-04-11 §
14:20 <andrewbogott> moving tools-bastion-mtemp to labvirt1009 [tools]
2016-04-06 §
15:20 <bd808> Removed local hack for T131906 from tools-puppetmaster-01 [tools]
2016-03-30 §
15:50 <andrewbogott> rebooting tools-proxy-01 in hopes of clearing some bad caches [tools]
2016-03-28 §
20:51 <yuvipanda> lifted RAM quota from 900Gigs to 1TB?! [tools]
20:30 <chasemp> change perm grant files from create-dbusers for chmod 400 chat chattr +i [tools]
2016-03-18 §
15:47 <chasemp> had to kill stalkboten as it was logging constant errors filling logs to the tune of hundreds of gigs [tools]
15:36 <chasemp> cleanup huge log collection for broken bot: /srv/project/tools/project/betacommand-dev/tspywiki/irc/logs# rm -fR SpamBotLog.log\.* [tools]
2016-03-11 §
20:57 <mutante> reverted font changes - puppet runs recovering [tools]
20:37 <mutante> more puppet issues due to font dependencies on trusty, on it [tools]
19:39 <mutante> should a tools-exec server be influenced by font packages on an mw appserver? [tools]
19:39 <mutante> fixed puppet runs on tools-exec (gerrit 276792) [tools]
2016-03-02 §
14:56 <chasemp> qdel 3956069 and 3758653 for abusing auth [tools]