| 2016-04-27
      
      § | 
    
  | 22:38 | <legoktm> | stop/started nodepool | [releng] | 
            
  | 22:36 | <thcipriani> | I don't have permission to restart nodepool | [releng] | 
            
  | 22:35 | <thcipriani> | restarting nodepool | [releng] | 
            
  | 22:18 | <matt_flaschen> | Deployed https://gerrit.wikimedia.org/r/#/c/282440/ to switch Beta Cluster to use External Store for new testwiki writes | [releng] | 
            
  | 20:59 | <hashar> | thcipriani downgraded git plugins successfully (we wanted to rule out their upgrade  for some weird issue) | [releng] | 
            
  | 20:13 | <cscott> | updated OCG to version e39e06570083877d5498da577758cf8d162c1af4 | [releng] | 
            
  | 14:10 | <hashar> | restarting Jenkins | [releng] | 
            
  | 14:09 | <hashar> | Jenkins upgrading credential plugin 1.24 > 1.27  And Credentials binding plugin 1.6 > 1.7 | [releng] | 
            
  | 14:07 | <hashar> | Jenkins upgrading git plugin 2.4.1 > 2.4.4 | [releng] | 
            
  | 14:01 | <hashar> | Jenkins upgrading git client plugin 1.19.1. > 1.19.6 | [releng] | 
            
  | 13:13 | <jzerebecki> | reloading zuul for 81a1f1a..0993349 | [releng] | 
            
  | 11:43 | <hashar> | fixed puppet on deployment-cache-text04  T132689 | [releng] | 
            
  | 10:38 | <hashar> | Rebuild Image ci-trusty-wikimedia-1461753210 in wmflabs-eqiad is ready | [releng] | 
            
  | 09:43 | <hashar> | tmh01.deployment-prep.eqiad.wmflabs denies mwdeploy user breaking https://integration.wikimedia.org/ci/job/beta-scap-eqiad/ | [releng] | 
            
  
    | 2016-04-26
      
      § | 
    
  | 20:45 | <hashar> | Regenerating Nodepool Jessie snapshot to include composer and HHVM | T128092 | [releng] | 
            
  | 20:23 | <jzerebecki> | reloading zuul for eb480d8..81a1f1a | [releng] | 
            
  | 19:25 | <jzerebecki> | reload zuul for 4675213..eb480d8 | [releng] | 
            
  | 19:25 | <jzerebecki> | 4675213..eb480d8 | [releng] | 
            
  | 14:18 | <hashar> | Applied security patches to 1.27.0-wmf.22 | T131556 | [releng] | 
            
  | 12:39 | <hashar> | starting cut of 1.27.0-wmf.22 branch ( poke ostriches ) | [releng] | 
            
  | 10:29 | <hashar> | restored integration/phpunit on CI slaves due to https://integration.wikimedia.org/ci/job/operations-mw-config-phpunit/ failling | [releng] | 
            
  | 09:11 | <hashar> | CI is back up! | [releng] | 
            
  | 08:20 | <hashar> | shutoff instance castor, does not seem to be able to start again :(  | T133652 | [releng] | 
            
  | 08:12 | <hashar> | hard rebooting castor instance | T133652 | [releng] | 
            
  | 08:10 | <hashar> | soft rebooting castor instance | T133652 | [releng] | 
            
  | 08:06 | <hashar> | CI jobs deadlocked due to castor being unavailable | https://phabricator.wikimedia.org/T133652 | [releng] | 
            
  | 00:46 | <thcipriani> | temporary keyholder fix in place in beta | [releng] | 
            
  | 00:18 | <thcipriani> | beta-scap-eqiad failure due to bad keyholder-auth.d fingerprints | [releng] | 
            
  
    | 2016-04-25
      
      § | 
    
  | 20:58 | <cscott> | updated OCG to version 58a720508deb368abfb7652e6a8c7225f95402d2 | [releng] | 
            
  | 19:46 | <hashar> | Nodepool now has a couple trusty instances intended to experiment with Zend 5.5 / HHVM migration . https://phabricator.wikimedia.org/T133203#2236625 | [releng] | 
            
  | 13:34 | <hashar> | Nodepool is attempting to create a Trusty snapshot with name ci-trusty-wikimedia-1461591203 | T133203 | [releng] | 
            
  | 13:15 | <hashar> | openstack image create --file /home/hashar/image-trusty-20160425T124552Z.qcow2 ci-trusty-wikimedia --disk-format qcow2 --property show=true  # T133203 | [releng] | 
            
  | 10:38 | <hashar> | Refreshing Nodepool Jessie snapshot based on new image | [releng] | 
            
  | 10:35 | <hashar> | Refreshed Nodepool Jessie image ( image-jessie-20160425T100035Z ) | [releng] | 
            
  | 09:24 | <hashar> | beta / scap failure filled as T133521 | [releng] | 
            
  | 09:20 | <hashar> | Keyholder / mwdeploy ssh keys have been messed up on beta cluster somehow :-( | [releng] | 
            
  | 08:47 | <hashar> | mwdeploy@deployment-tin has lost ssh host keys file :( | [releng] |