| 
      
        2015-05-27
      
      §
     | 
  
    
  | 21:15 | 
  <AaronSchulz> | 
  populated jobqueue:aggregator:s-wikis:v2 with 1000 fake wiki keys for load testing | 
  [releng] | 
            
  | 21:07 | 
  <AaronSchulz> | 
  Deployed https://gerrit.wikimedia.org/r/#/c/208852/ | 
  [releng] | 
            
  | 21:07 | 
  <AaronSchulz> | 
  Deleted 4G of logs on jobrunner01 | 
  [releng] | 
            
  | 15:09 | 
  <hashar> | 
  Jenkins slaves are all back up. Root cause was some ssh algorithm in their sshd which is not supported by Jenkins jsch embedded lib. | 
  [releng] | 
            
  | 14:30 | 
  <hashar> | 
  manually rebasing puppet git on deployment-salt (stalled) | 
  [releng] | 
            
  | 14:27 | 
  <hashar> | 
  restarting deployment-salt / some process is 100% wa/IO | 
  [releng] | 
            
  | 13:38 | 
  <hashar> | 
  restarted integration puppetmaster (memory leak) | 
  [releng] | 
            
  | 13:35 | 
  <hashar> | 
  integration-puppetmaster apparently out of memory  | 
  [releng] | 
            
  | 13:30 | 
  <hashar> | 
  All Jenkins slaves are disconnected due to some ssh error. CI is down. | 
  [releng] |