3101-3150 of 4541 results (13ms)
2014-06-16 §
20:34 <bd808> Puppet disabled on deployment-jobrunner01 since 2014-06-03; No SAL logs explaining why [releng]
20:19 <bd808> Updated scap to 5adce72; trebuchet reported i-00000237 (deployment-videoscaler01) as not updating, but manual check shows it did sync properly [releng]
20:00 <bd808> Deleted /var/lib/puppet/state/agent_catalog_run.lock on deployment-bastion after verifying that no puppet processes were running [releng]
19:55 <bd808> Truncated /var/log/diamond/diamond.log and restarted diamond on deployment-bastion [releng]
19:36 <bd808> /var/log/diamond is 787M of 1.2G total logs [releng]
19:29 <bd808> /var 0% free on deployment-bastion; looking for things to clean-up [releng]
2014-06-09 §
15:19 <andrewbogott> doing a 'rebase origin' on deployment-salt, because it needs it. [releng]
15:10 <andrewbogott> updating all instances to puppet 3 via a cherry-pick�� of https://gerrit.wikimedia.org/r/#/c/137898/ on deployment-salt [releng]
2014-06-07 §
02:44 <bd808> Restarted logstash on deployment-logstash1; last even logged at 2014-06-06T22:11:04 [releng]
2014-06-06 §
19:26 <bblack> - synced labs/private on deployment-salt again [releng]
16:30 <bd808> Rebooted deployment-salt [releng]
16:27 <bd808> Made /var/log a symlink to /srv/var-log on deployment-salt [releng]
16:26 <bblack> Updated labs/private.git on puppetmaster. brings in updated zero+netmapper password for beta [releng]
16:18 <bd808> Changed from role::labs::lvm::biglogs to role::labs::lvm::srv on deployment-salt and made /var/lib a symlink to /srv/var-lib [releng]
15:45 <bd808> /var on deployment-salt still at 97% full after moving logs; /var/lib is our problem [releng]
15:43 <bd808> Archived deployment-salt:/var/log to /data/project/deployment-salt [releng]
15:40 <bd808> Disabled puppet on deployment-salt to work on disk space issues [releng]
12:44 <hashar> Updated labs/private.git on puppetmaster. Brings Brandon Black change "add labs copy of zerofetcher auth file" {{gerrit|137918}} [releng]
02:48 <mwalker> added role::labs::lvm::biglogs to deployment-salt because it is out of room on /var and I don't know what I can delete [releng]
01:25 <bd808> Live hacked /etc/apache2/wmf/hhvm.conf on apaches to allow them to start [releng]
00:30 <bd808> `git stash`ed dirty dblist files found in /a/common on deployment-bastion [releng]
2014-06-05 §
14:16 <manybubbles> rebuild beta's jawiki's search index without kuromoji - it didn't help much anyway [releng]
14:01 <manybubbles> elasticsearch cluster got really angry in beta when I restarted some node - its like they aren't talking to eachother properly - trying to recover. once that is done I'll upgrade to 1.2.1 and that might fix it [releng]
13:59 <hashar> deployment-elastic01 puppet was broken due to {{bug|63322}} i.e. having some HTML garbage as ec2id which would be used as puppet certname [releng]
13:47 <manybubbles> rolling restart of elasticsearch nodes in beta to pick up new kernel [releng]
2014-06-04 §
20:46 <bd808> Fixed file ownership on /data/project/apache/uncommon for beta-recompile-math-texvc-eqiad job [releng]
19:27 <manybubbles> sorry, can't do that yet, [releng]
19:27 <manybubbles> plugins deployed to beta - time to restart Elasticsearch in beta - should cause not interruption of service [releng]
19:01 <manybubbles> deploying Elasticsearch 1.2.1 and some updated plugins to beta [releng]
17:11 <bd808> Unwedged the jenkins jobs to updating beta by stopping the stuck db update job [releng]
16:27 <bd808> Changed uid/git for files owned by l10nupdate user [releng]
09:50 <mwalker> Reset salt caches by running `salt '*' state.clear_cache` from deployment-salt -- deployment-pdf01 now no longer reports errors when returning status for deployment [releng]
2014-06-03 §
22:30 <bd808> Deleted unused /data/project/apache/common-local on NFS share. [releng]
2014-06-02 §
19:42 <bd808> Updated scap to a7da355 [releng]
05:14 <bd808> Restarted logstash on deployment-logstash1; Last event logged at 2014-06-01T0722:56 [releng]
2014-05-30 §
21:45 <bd808> Restarted uwsgi on deployment-graphite [releng]
18:43 <bd808> Updated scap to c4204dd [releng]
2014-05-29 §
21:07 <bd808> mwalker cleaned up log spam from upstart on deployment-pdf01 [releng]
20:59 <bd808> /var full on deployment-pdf01 [releng]
20:55 <bd808> Restarted salt minion on deployment-pdf01 with `sudo salt 'i-00000396.eqiad.wmflabs' service.restart salt-minion` [releng]
2014-05-28 §
17:53 <bd808> Restarted logstash on deployment-logstash1; last event logged at 2014-05-28T12:11:37 [releng]
16:56 <bd808> Updated scap to fd7e538 [releng]
2014-05-27 §
19:08 <bd808> Updated scap to 48c7e28 [releng]
14:56 <bd808> Updated scap to 9609e8d [releng]
2014-05-23 §
16:32 <bd808> Upgraded elasticsearch to 1.1.0 on deployment-logstash1 [releng]
13:36 <manybubbles> restarting elasticsearch on deployment-elastic01 to pick up some gc setting recommended by elasticsearch team [releng]
2014-05-22 §
23:00 <bd808> Added 20after4 as a project admin [releng]
22:59 <bd808> Added matanya as a project memeber [releng]
21:38 <bd808|LUNCH> Deployed scap 096cb3f [releng]
2014-05-21 §
17:33 <mwalker> converted deployment-pdf01 (i-00000396.eqiad.wmflabs) to use local puppet & salt master [releng]