| 
      
        2014-09-30
      
      §
     | 
  
    
  | 15:49 | 
  <bd808> | 
  apt-get dist-upgrade fixed hhvm on deployment-mediawiki03 | 
  [releng] | 
            
  | 15:45 | 
  <hashar> | 
  Updating our Jenkins job builder fork  686265a..ee80dbc (no job changed) | 
  [releng] | 
            
  | 15:44 | 
  <bd808> | 
  scap failing in beta due to "Permission denied (publickey)" talking to deployment-rsync01.eqiad.wmflabs | 
  [releng] | 
            
  | 15:39 | 
  <bd808> | 
  hhvm not starting after puppet run on deployment-mediawiki03. Investigating. | 
  [releng] | 
            
  | 15:36 | 
  <bd808> | 
  enabling puppet and forcing run on deployment-mediawiki03 | 
  [releng] | 
            
  | 15:36 | 
  <bd808> | 
  enabling puppet and forcing run on deployment-mediawiki03 | 
  [releng] | 
            
  | 15:34 | 
  <bd808> | 
  enabling puppet and forcing run on deployment-mediawiki02 | 
  [releng] | 
            
  | 15:34 | 
  <bd808> | 
  enabling puppet and forcing run on deployment-mediawiki02 | 
  [releng] | 
            
  | 15:29 | 
  <bd808> | 
  puppet showed no changes on mediawiki01‽ | 
  [releng] | 
            
  | 15:28 | 
  <bd808> | 
  enabling puppet and forcing run on deployment-mediawiki01 | 
  [releng] | 
            
  | 15:27 | 
  <bd808> | 
  enabling puppet and forcing run on deployment-mediawiki01 | 
  [releng] | 
            
  | 15:13 | 
  <bd808> | 
  Fixed logstash by installing http://packages.elasticsearch.org/logstash/1.4/debian/pool/main/l/logstash-contrib/logstash-contrib_1.4.2-1-efd53ef_all.deb | 
  [releng] | 
            
  | 15:02 | 
  <bd808> | 
  Logstash doesn't bundle the prune filter by default any more -- http://logstash.net/docs/1.4.2/filters/prune | 
  [releng] | 
            
  | 14:59 | 
  <bd808> | 
  Logstash rules need to be adjusted for latest upstream version: "Couldn't find any filter plugin named 'prune'" | 
  [releng] | 
            
  | 12:37 | 
  <hashar> | 
  Fixed some file permissions under deployment-bastion:/srv/mediawiki-staging/php-master/vendor/.git  some files belonged to root instead of mwdeploy | 
  [releng] | 
            
  | 00:34 | 
  <bd808> | 
  Updated kibana to latest upstream head 8653aba | 
  [releng] | 
            
  
    | 
      
        2014-09-29
      
      §
     | 
  
    
  | 22:45 | 
  <Reedy> | 
  re-enabled beta-scap-eqiad | 
  [releng] | 
            
  | 21:34 | 
  <Reedy> | 
  disabled "beta-scap-eqiad" until things are fixed | 
  [releng] | 
            
  | 21:24 | 
  <Reedy> | 
  deleted l10n cache on deployment-rsync01 to attempt to run sync-common manually | 
  [releng] | 
            
  | 21:22 | 
  <Reedy> | 
  deployment-rsync01 hard drive is far too small | 
  [releng] | 
            
  | 17:57 | 
  <cscott> | 
  updated OCG to version 89d8f29a24295b05d0643abe976fea83b56575c9 | 
  [releng] | 
            
  | 14:22 | 
  <hashar> | 
  apt-get upgrade and reboot of all integration-slaveXX instances | 
  [releng] | 
            
  | 14:07 | 
  <hashar> | 
  updated puppetmaster labs/private on both integration and beta cluster projects ( a41fcdd..84f0906 ) | 
  [releng] | 
            
  | 08:57 | 
  <hashar> | 
  rebased puppetmaster | 
  [releng] | 
            
  | 06:58 | 
  <ori> | 
  Configured Beta cluster to use redis for session storage | 
  [releng] | 
            
  | 06:57 | 
  <ori> | 
  Created deployment-redis02 and converted it to use local puppet & salt masters | 
  [releng] | 
            
  | 05:23 | 
  <ori> | 
  Created deployment-redis01 and converted it to use local puppet & salt masters | 
  [releng] | 
            
  
    | 
      
        2014-09-25
      
      §
     | 
  
    
  | 23:35 | 
  <bd808> | 
  Done messing with puppet repo. Replaced 2 local commits with proper gerrit cherry picks. Removed a cherry-pick that had been rearranged and merged. Removed a cherry-pick that had been abandoned in gerrit. | 
  [releng] | 
            
  | 23:10 | 
  <bd808> | 
  removed cherry-pick of abandoned https://gerrit.wikimedia.org/r/#/c/156223/; if beta wikis stop working this would be a likely culprit | 
  [releng] | 
            
  | 22:48 | 
  <RoanKattouw> | 
  Fixed permissions of deployment-bastion:/srv/deployment/mathoid/mathoid/.git/deploy (needed g+w) | 
  [releng] | 
            
  | 22:36 | 
  <bd808> | 
  Trying to reduce the number of untracked changes in puppet repo. Expect some short term breakage. | 
  [releng] | 
            
  | 22:21 | 
  <bd808> | 
  cleaned up puppet repo with `git rebase origin/production; git submodule update --init --recursive` | 
  [releng] | 
            
  | 22:18 | 
  <bd808> | 
  puppet repo on deployment-salt out of whack. I will try to fix. | 
  [releng] | 
            
  | 11:36 | 
  <_joe_> | 
  updated hhvm to fix most bugs, also cherry-picked https://gerrit.wikimedia.org/r/#/c/162839/ | 
  [releng] | 
            
  | 08:15 | 
  <hashar> | 
  beta: puppetmaster rebased | 
  [releng] | 
            
  | 08:10 | 
  <hashar> | 
  beta: dropped a patch that reverted OCG LVS configuration ( https://gerrit.wikimedia.org/r/#/c/146860/ ), it has been fixed by https://gerrit.wikimedia.org/r/#/c/148371/ | 
  [releng] | 
            
  | 08:04 | 
  <hashar> | 
  attempting to rebase beta cluster puppet master. Currently at 74036376  | 
  [releng] |