2151-2200 of 3577 results (5ms)
2014-06-06 §
15:45 <bd808> /var on deployment-salt still at 97% full after moving logs; /var/lib is our problem [releng]
15:43 <bd808> Archived deployment-salt:/var/log to /data/project/deployment-salt [releng]
15:40 <bd808> Disabled puppet on deployment-salt to work on disk space issues [releng]
12:44 <hashar> Updated labs/private.git on puppetmaster. Brings Brandon Black change "add labs copy of zerofetcher auth file" {{gerrit|137918}} [releng]
02:48 <mwalker> added role::labs::lvm::biglogs to deployment-salt because it is out of room on /var and I don't know what I can delete [releng]
01:25 <bd808> Live hacked /etc/apache2/wmf/hhvm.conf on apaches to allow them to start [releng]
00:30 <bd808> `git stash`ed dirty dblist files found in /a/common on deployment-bastion [releng]
2014-06-05 §
14:16 <manybubbles> rebuild beta's jawiki's search index without kuromoji - it didn't help much anyway [releng]
14:01 <manybubbles> elasticsearch cluster got really angry in beta when I restarted some node - its like they aren't talking to eachother properly - trying to recover. once that is done I'll upgrade to 1.2.1 and that might fix it [releng]
13:59 <hashar> deployment-elastic01 puppet was broken due to {{bug|63322}} i.e. having some HTML garbage as ec2id which would be used as puppet certname [releng]
13:47 <manybubbles> rolling restart of elasticsearch nodes in beta to pick up new kernel [releng]
2014-06-04 §
20:46 <bd808> Fixed file ownership on /data/project/apache/uncommon for beta-recompile-math-texvc-eqiad job [releng]
19:27 <manybubbles> sorry, can't do that yet, [releng]
19:27 <manybubbles> plugins deployed to beta - time to restart Elasticsearch in beta - should cause not interruption of service [releng]
19:01 <manybubbles> deploying Elasticsearch 1.2.1 and some updated plugins to beta [releng]
17:11 <bd808> Unwedged the jenkins jobs to updating beta by stopping the stuck db update job [releng]
16:27 <bd808> Changed uid/git for files owned by l10nupdate user [releng]
09:50 <mwalker> Reset salt caches by running `salt '*' state.clear_cache` from deployment-salt -- deployment-pdf01 now no longer reports errors when returning status for deployment [releng]
2014-06-03 §
22:30 <bd808> Deleted unused /data/project/apache/common-local on NFS share. [releng]
2014-06-02 §
19:42 <bd808> Updated scap to a7da355 [releng]
05:14 <bd808> Restarted logstash on deployment-logstash1; Last event logged at 2014-06-01T0722:56 [releng]
2014-05-30 §
21:45 <bd808> Restarted uwsgi on deployment-graphite [releng]
18:43 <bd808> Updated scap to c4204dd [releng]
2014-05-29 §
21:07 <bd808> mwalker cleaned up log spam from upstart on deployment-pdf01 [releng]
20:59 <bd808> /var full on deployment-pdf01 [releng]
20:55 <bd808> Restarted salt minion on deployment-pdf01 with `sudo salt 'i-00000396.eqiad.wmflabs' service.restart salt-minion` [releng]
2014-05-28 §
17:53 <bd808> Restarted logstash on deployment-logstash1; last event logged at 2014-05-28T12:11:37 [releng]
16:56 <bd808> Updated scap to fd7e538 [releng]
2014-05-27 §
19:08 <bd808> Updated scap to 48c7e28 [releng]
14:56 <bd808> Updated scap to 9609e8d [releng]
2014-05-23 §
16:32 <bd808> Upgraded elasticsearch to 1.1.0 on deployment-logstash1 [releng]
13:36 <manybubbles> restarting elasticsearch on deployment-elastic01 to pick up some gc setting recommended by elasticsearch team [releng]
2014-05-22 §
23:00 <bd808> Added 20after4 as a project admin [releng]
22:59 <bd808> Added matanya as a project memeber [releng]
21:38 <bd808|LUNCH> Deployed scap 096cb3f [releng]
2014-05-21 §
17:33 <mwalker> converted deployment-pdf01 (i-00000396.eqiad.wmflabs) to use local puppet & salt master [releng]
14:50 <bd808> restarted logstash on deployment-logstash1; getting really tired of these soft crashes [releng]
00:33 <bd808> Puppet failing on deployment-videoscaler01 with duplicate definition of Class[Mediawiki::Jobrunner] [releng]
00:07 <bd808> Fixed puppet for deployment-jobrunner01 using https://gerrit.wikimedia.org/r/#/c/134519/2 [releng]
2014-05-20 §
23:49 <bd808> Fixed puppet for deployment-apache[12] using https://gerrit.wikimedia.org/r/#/c/134519/2 [releng]
23:11 <bd808> deployment-apache01 needs more work: "Could not set shell on user[mwdeploy]" [releng]
23:06 <bd808> Fixing puppet config for upstream rename of role::applicationserver -> role::mediawiki [releng]
21:14 <ori> Converted deployment-stream to use local puppet & salt masters [releng]
21:08 <RoanKattouw> chown'ed /data/project/parsoid/parsoid.log from mwalker (?!?) to parsoid so Parsoid runs again [releng]
15:53 <bd808> Deployed scap 7b6fc47 via trebuchet [releng]
2014-05-19 §
14:34 <bd808> Restarted logstash service on deployment-logstash1; it stopped logging new events at 10:37:13Z [releng]
2014-05-16 §
21:20 <manybubbles> restarting elasticsearch in beta to update some plugins [releng]
00:34 <bd808> Updated EventLogging to I89819bd [releng]
2014-05-15 §
22:14 <bd808> Restarted logstash on deployment-logstash1 yet again; memory leak from invalid encoding bug [releng]
00:14 <bd808> Disabled puppet on deployment-logstash1 to test a local logstash config change [releng]