| 
      
        2015-04-29
      
      §
     | 
  
    
  | 21:03 | 
  <andrewbogott> | 
  suspending and shrinking disks of many instances | 
  [releng] | 
            
  | 17:15 | 
  <thcipriani> | 
  removed l10nupdate user from /etc/passwd on deployment-bastion | 
  [releng] | 
            
  | 15:00 | 
  <hashar> | 
  Instances are being moved out from labvirt1005 which has some faulty memory. List of instances at https://phabricator.wikimedia.org/T97521#1245217 | 
  [releng] | 
            
  | 14:25 | 
  <hashar> | 
  upgrading zuul on integration-slave-precise-1011 for  https://phabricator.wikimedia.org/T97106 | 
  [releng] | 
            
  | 14:11 | 
  <hashar> | 
  rebooting integration-saltmaster stalled. | 
  [releng] | 
            
  | 13:11 | 
  <hashar> | 
  Rebooting deployment-parsoid05 via wikitech interface. | 
  [releng] | 
            
  | 13:02 | 
  <hashar> | 
  labvirt1005 seems to have hardware issue. Impacts a bunch of beta cluster / integration instances as listed on https://phabricator.wikimedia.org/T97521#1245217 | 
  [releng] | 
            
  | 12:22 | 
  <hashar> | 
  deployment-parsoid05 slow down is https://phabricator.wikimedia.org/T97421  . Running apt-get upgrade and rebooting it but its slowness issue might be with the underlying hardware | 
  [releng] | 
            
  | 12:13 | 
  <hashar> | 
  killing puppet on deployment-parsoid05   eats all CPU for some reason | 
  [releng] | 
            
  | 02:40 | 
  <legoktm> | 
  deploying https://gerrit.wikimedia.org/r/207363 and https://gerrit.wikimedia.org/r/207368 | 
  [releng] | 
            
  
    | 
      
        2015-04-28
      
      §
     | 
  
    
  | 23:33 | 
  <hoo> | 
  Ran foreachwiki extensions/Wikidata/extensions/Wikibase/lib/maintenance/populateSitesTable.php --load-from 'http://meta.wikimedia.beta.wmflabs.org/w/api.php' to fix all sites tables | 
  [releng] | 
            
  | 23:18 | 
  <hoo> | 
  Ran mysql> INSERT INTO sites (SELECT * FROM wikidatawiki.sites); on enwikinews to populate the sites table | 
  [releng] | 
            
  | 23:18 | 
  <hoo> | 
  Ran mysql> INSERT INTO sites (SELECT * FROM wikidatawiki.sites); on testwiki to populate the sites table | 
  [releng] | 
            
  | 20:57 | 
  <YuviPanda> | 
  KILL KILL KILL DEPLOYMENT-LUCID-SALT WITH FIRE AND BRIMSTONE AND BAD THINGS | 
  [releng] | 
            
  | 17:48 | 
  <James_F> | 
  Restarting grrrit-wm for config change. | 
  [releng] | 
            
  | 16:24 | 
  <bd808> | 
  Updated scap to ef15380 (Make scap localization cache build $TMPDIR aware) | 
  [releng] | 
            
  | 15:42 | 
  <bd808> | 
  Freed 5G on deployment-bastion by deleting abandoned /tmp/scap_l10n_* directories | 
  [releng] | 
            
  | 14:01 | 
  <marxarelli> | 
  reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/206967/ | 
  [releng] | 
            
  | 00:17 | 
  <greg-g> | 
  after the 3rd or so time doing it (while on the Golden Gate Bridge, btw) it worked | 
  [releng] | 
            
  | 00:11 | 
  <greg-g> | 
  still nothing... | 
  [releng] | 
            
  | 00:10 | 
  <greg-g> | 
  after disconnecting, marking temp offline, bringing back online, and launching slave agent: "Slave successfully connected and online" | 
  [releng] | 
            
  | 00:07 | 
  <greg-g> | 
  deployment-bastion is idle, yet we have 3 pending jobs waiting for an executer on it - will disconnect/reconnect it in Jenkins | 
  [releng] | 
            
  
    | 
      
        2015-04-27
      
      §
     | 
  
    
  | 21:45 | 
  <bd808> | 
  Manually triggered beta-mediawiki-config-update-eqiad for zuul build df1e789c726ad4aae60d7676e8a4fc8a2f6841fb | 
  [releng] | 
            
  | 21:20 | 
  <bd808> | 
  beta-scap-equad job green again after adding a /srv/ disk to deployment-jobrunner01 | 
  [releng] | 
            
  | 21:08 | 
  <bd808> | 
  Applied role::labs::lvm::srv on deployment-jobrunner01 and forced puppet run | 
  [releng] | 
            
  | 21:08 | 
  <bd808> | 
  Deleted deployment-jobrunner01:/srv/* in preparation for applying role::labs::lvm::srv | 
  [releng] | 
            
  | 21:06 | 
  <bd808> | 
  deployment-jobrunner01 missing role::labs::lvm::srv | 
  [releng] | 
            
  | 21:00 | 
  <bd808> | 
  Root partition full on deployment-jobrunner01 | 
  [releng] | 
            
  | 20:53 | 
  <bd808> | 
  removed mwdeploy user from deployment-bastion:/etc/passwd | 
  [releng] | 
            
  | 20:15 | 
  <Krinkle> | 
  Relaunched Gearman connection | 
  [releng] | 
            
  | 19:53 | 
  <Krinkle> | 
  Jenkins unable to re-create Gearman connection. (HTTP 503 error from /configure). Have to force restart Jenkins | 
  [releng] |