| 2015-07-24
      
      § | 
    
  | 17:17 | <marxarelli> | running `jenkins-jobs update config/ 'mediawiki-selenium-integration' 'mwext-mw-selenium'` to deploy Ib289d784c7b3985bd4823d967fbc07d5759dc756 | [releng] | 
            
  | 17:05 | <marxarelli> | running `jenkins-jobs update config/ 'mediawiki-selenium-integration'` to deploy and test Ib289d784c7b3985bd4823d967fbc07d5759dc756 | [releng] | 
            
  | 17:04 | <hashar> | integration-saltmaster, in  a '''screen''' : salt -b 1 '*slave*' cmd.run '/usr/local/sbin/puppet-run'|tee hashar-massrun.log | [releng] | 
            
  | 17:04 | <hashar> | cancelled last command | [releng] | 
            
  | 17:03 | <hashar> | integration-saltmaster : salt -b 1 '*slave*' cmd.run '/usr/local/sbin/puppet-run' &  && disown && exit | [releng] | 
            
  | 16:55 | <hashar> | Might have fixed the puppet/pip mess on CI slaves by creating a symlink from /usr/bin/pip to /usr/local/bin/pip ( https://gerrit.wikimedia.org/r/#/c/226729/1..2/modules/contint/manifests/packages/python.pp,unified ) | [releng] | 
            
  | 16:36 | <hashar> | puppet on Jenkins slaves might have some intermittent issues due to pip installation  https://gerrit.wikimedia.org/r/226729 | [releng] | 
            
  | 15:29 | <hashar> | removing pip obsolete download-cache setting ( https://gerrit.wikimedia.org/r/#/c/226730/ ) | [releng] | 
            
  | 15:27 | <hashar> | upgrading pip to 7.1.0 via pypi ( https://gerrit.wikimedia.org/r/#/c/226729/ ).  Revert plan is to uncherry pick the patch on the puppetmaster and:  pip uninstall pip | [releng] | 
            
  | 12:46 | <hashar> | Jenkins: switching gearman plugin from our custom compiled 0.1.1-9-g08e9c42-change_192429_2  to upstream 0.1.2. They are actually the exact same versions. | [releng] | 
            
  | 08:40 | <hashar> | upgrading zuul to zuul_2.0.0-327-g3ebedde-wmf3precise1 to fix a regression ( https://phabricator.wikimedia.org/T106531 ) | [releng] | 
            
  | 08:39 | <hashar> | upgrading zuul | [releng] | 
            
  
    | 2015-07-23
      
      § | 
    
  | 23:03 | <marxarelli> | running `jenkins-jobs update config/ 'browsertests-*'` to deploy I2d0f83d0c6a406d46627578cb8db0706d1b8655d | [releng] | 
            
  | 16:38 | <marxarelli> | Reloading Zuul to deploy I96b6218a208f133209452c71bcf01a1088305aea | [releng] | 
            
  | 15:40 | <urandom> | applied wip logstash & cassandra changes (https://gerrit.wikimedia.org/r/#/c/226025/) to deployment-prep | [releng] | 
            
  | 13:24 | <hashar> | apt-get upgrade integration-puppetmaster and rebooting it | [releng] | 
            
  | 13:23 | <hashar> | integration puppetmaster in bad shape: Warning: Error 400 on SERVER: Cannot allocate memory - fork(2) | [releng] | 
            
  | 10:58 | <hashar> | beta : salt '*' cmd.run 'rm /etc/apt/apt.conf.d/20auto-upgrades.ucf-dist' | [releng] | 
            
  | 10:52 | <hashar> | Beta cluster puppetmaster is now deployment-puppetmaster.deployment-prep.eqiad.wmflabs . Migrated all instances (solves https://phabricator.wikimedia.org/T106649 ) | [releng] | 
            
  | 10:30 | <hashar> | regenerated puppet cert on deployment-salt , the old puppetmaster now a puppet client | [releng] | 
            
  | 10:23 | <hashar> | running apt-get upgrade on deployment-parsoidcache02 | [releng] | 
            
  | 09:32 | <hashar> | puppet broken on deployment-fluorine : Error: Could not request certificate: Neither PUB key nor PRIV key:: header too long | [releng] | 
            
  | 08:39 | <hashar> | Disabling puppet agent on ALL beta cluster instances | [releng] | 
            
  | 08:18 | <hashar> | creating deployment-puppetmaster m1.medium :D | [releng] | 
            
  | 01:57 | <jzerebecki> | reconnected slave and needed to kill a few pending beta jobs, works again | [releng] | 
            
  | 01:50 | <jzerebecki> | trying https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update | [releng] | 
            
  | 01:09 | <legoktm> | beta-mediawiki-config-update-eqiad jobs stuck | [releng] | 
            
  | 00:15 | <jzerebecki> | reloading zuul 369e6eb..73dc1f6 for https://gerrit.wikimedia.org/r/#/c/223527/ | [releng] | 
            
  
    | 2015-07-21
      
      § | 
    
  | 22:54 | <greg-g> | 22:50 <   chasemp> "then git reset --hard 9588d0a6844fc9cc68372f4bf3e1eda3cffc8138 in  /etc/zuul/wikimedia" | [releng] | 
            
  | 22:53 | <greg-g> | 22:47 <   chasemp> service zuul stop && service zuul-merger stop && sudo apt-get install  zuul=2.0.0-304-g685ca22-wmf1precise1 | [releng] | 
            
  | 21:48 | <greg-g> | Zuul not responding | [releng] | 
            
  | 20:23 | <hasharConfcall> | Zuul no more reports back to Gerrit due to an error with the Gerrit label | [releng] | 
            
  | 20:10 | <hasharConfcall> | Zuul restarted with 2.0.0-327-g3ebedde-wmf2precise1 | [releng] | 
            
  | 19:48 | <hasharConfcall> | Upgrading Zuul to zuul_2.0.0-327-g3ebedde-wmf2precise1  Previous version failed because python-daemon was too old, now shipped in the venv  https://phabricator.wikimedia.org/T106399 | [releng] | 
            
  | 15:04 | <hashar> | upgraded Zuul on gallium from zuul_2.0.0-306-g5984adc-wmf1precise1_amd64.deb to zuul_2.0.0-327-g3ebedde-wmf1precise1_amd64.deb . now uses python-daemon 2.0.5 | [releng] | 
            
  | 13:37 | <hashar> | upgraded Zuul on gallium from zuul_2.0.0-304-g685ca22-wmf1precise1 to zuul_2.0.0-306-g5984adc-wmf1precise1 . Uses a new version of GitPython | [releng] | 
            
  | 02:15 | <bd808> | upgraded to elasticsearch-1.7.0.deb on deployment-logstash2 | [releng] |