| 
      
        2015-04-02
      
      §
     | 
  
    
  | 14:04 | 
  <hashar> | 
  uninstall the pip installed zuul version from Precise labs slaves by doing:  pip uninstall zuul && rm /usr/local/bin/zuul* . Switching them all to a Debian package | 
  [releng] | 
            
  | 13:45 | 
  <hashar> | 
  pooling back integration-slave1001 and 1002 which are using zuul-cloner provided by a debian package | 
  [releng] | 
            
  | 13:35 | 
  <hashar> | 
  reloading Jenkins configuration files from disk to make it knows about a change manually applied to most jobs config.xml files for https://gerrit.wikimedia.org/r/#/c/201451/ | 
  [releng] | 
            
  | 13:01 | 
  <Krinkle> | 
  Reloading Zuul to deploy https://gerrit.wikimedia.org/r/201458  | 
  [releng] | 
            
  | 12:19 | 
  <hashar> | 
  preventing job to run on integration-slave1001 by replacing its label with 'DoNotLabelThisSlaveHashar'. Going to install Zuul debian package on it | 
  [releng] | 
            
  | 09:37 | 
  <hashar> | 
  rebooting integration-zuul-server   homedir seems to be stalled/missing | 
  [releng] | 
            
  | 08:12 | 
  <hashar> | 
  upgrading packages on integration-dev | 
  [releng] | 
            
  | 05:14 | 
  <greg-g> | 
  and right when I log'd that, things seem to be recovering | 
  [releng] | 
            
  | 05:12 | 
  <greg-g> | 
  the shinken alerts about beta cluster issues are due to wmflabs having issues. | 
  [releng] | 
            
  
    | 
      
        2015-04-01
      
      §
     | 
  
    
  | 07:17 | 
  <Krinkle> | 
  Creating integration-slave1410 as test. Will re-create our pool later today. | 
  [releng] | 
            
  | 06:26 | 
  <Krinkle> | 
  Apply puppetmaster::autosigner to integration-puppetmaster | 
  [releng] | 
            
  | 05:51 | 
  <legoktm> | 
  deleting non-existent job workspaces from integration slaves | 
  [releng] | 
            
  | 05:42 | 
  <Krinkle> | 
  Free up space on integration-slave1001-1004 by removing obsolete phplint and qunit workspaces | 
  [releng] | 
            
  | 02:05 | 
  <Krinkle> | 
  Restarting Jenkins again.. | 
  [releng] | 
            
  | 01:35 | 
  <legoktm> | 
  started zuul on gallium | 
  [releng] | 
            
  | 01:00 | 
  <Krinkle> | 
  Restarting Jenkins | 
  [releng] | 
            
  | 01:00 | 
  <Krinkle> | 
  Jenkins is unable to start Gearman connection (HTTP 503);  | 
  [releng] | 
            
  | 01:00 | 
  <Krinkle> | 
  Force restarted Zuul, didn't help | 
  [releng] | 
            
  | 00:55 | 
  <Krinkle> | 
  Jenkins stuck. Builds are queued in Zuul but nothing is sent to Jenkins. | 
  [releng] | 
            
  
    | 
      
        2015-03-30
      
      §
     | 
  
    
  | 22:58 | 
  <legoktm> | 
  1001-1003 were depooled, restarted and repooled. 1004 is depooled and restarted | 
  [releng] | 
            
  | 22:40 | 
  <legoktm> | 
  rebooting precise jenkins slaves | 
  [releng] | 
            
  | 22:33 | 
  <Josve05a> | 
  manually start mysql on db1 and db2 | 
  [releng] | 
            
  | 21:57 | 
  <YuviPanda> | 
  reboot all instances from virt1000 | 
  [releng] | 
            
  | 21:40 | 
  <greg-g> | 
  Beta Cluster is down due to WMF Labs issues, being taken care of now (by Coren and Yuvi) | 
  [releng] | 
            
  | 19:53 | 
  <legoktm> | 
  deleted core dumps from integration-slave1001 | 
  [releng] | 
            
  | 19:11 | 
  <legoktm> | 
  deploying https://gerrit.wikimedia.org/r/200646 | 
  [releng] | 
            
  | 16:29 | 
  <jzerebecki> | 
  another damaged git repo integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-qunit/src/vendor/ | 
  [releng] | 
            
  | 16:07 | 
  <jzerebecki> | 
  removing workspaces of deleted jobs integration-slave100{1,2,3,4}:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-{client,repo,repo-api}-tests{,@*} | 
  [releng] | 
            
  | 15:14 | 
  <jzerebecki> | 
  integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-repo-api-tests-sqlite | 
  [releng] | 
            
  | 15:05 | 
  <jzerebecki> | 
  integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-repo-api-tests-mysql/src/extensions/cldr | 
  [releng] | 
            
  | 14:36 | 
  <jzerebecki> | 
  integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-*-tests{,@*} | 
  [releng] | 
            
  | 13:06 | 
  <jzerebecki> | 
  integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-client-tests@* | 
  [releng] | 
            
  | 13:05 | 
  <jzerebecki> | 
  integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-client-tests | 
  [releng] |