| 
      
        2014-12-18
      
      §
     | 
  
    
  | 23:56 | 
  <bd808> | 
  killed some ancient screen sessions on deployment-bastion | 
  [releng] | 
            
  | 23:53 | 
  <bd808> | 
  Restarted udp2log-mw on deployment-bastion | 
  [releng] | 
            
  | 23:53 | 
  <bd808> | 
  Restarted salt-minion on deployement-bastion | 
  [releng] | 
            
  | 23:47 | 
  <bd808> | 
  Updated scap to latest HEAD version | 
  [releng] | 
            
  | 21:57 | 
  <Krinkle> | 
  integration-slave1005 is not ready. It's incompletely setup due to https://phabricator.wikimedia.org/T84917 | 
  [releng] | 
            
  | 19:29 | 
  <marxarelli> | 
  restarted puppetmaster on deployment-salt | 
  [releng] | 
            
  | 19:29 | 
  <marxarelli> | 
  seeing "Could not evaluate: getaddrinfo: Temporary failure in name resolution" in the deployment-* puppet logs | 
  [releng] | 
            
  | 14:17 | 
  <hashar> | 
  deleting instance deployment-parsoid04 and removing it from Jenkins | 
  [releng] | 
            
  | 14:08 | 
  <hashar> | 
  restarted varnish backend on parsoidcache02 | 
  [releng] | 
            
  | 14:00 | 
  <hashar> | 
  parsoid05 seems happy: curl http://localhost:8000/_version: <tt>{"name":"parsoid","version":"0.2.0-git","sha":"d16dd2db6b3ca56e73439e169d52258214f0aeb2"}</tt> | 
  [releng] | 
            
  | 14:00 | 
  <hashar> | 
  parsoid05 seems happy: curl http://localhost:8000/_version<br/> | 
  [releng] | 
            
  | 13:56 | 
  <hashar> | 
  applying latest changes of Parsoid on parsoid05 via: <tt>zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/services/parsoid --change 180671,2</tt> | 
  [releng] | 
            
  | 13:56 | 
  <hashar> | 
  parsoid05: disabling puppet, stopping parsoid, rm -fR /srv/deployment/parsoid ; rerunning the Jenkins beta-parsoid-update-eqiad to hopefully recreate everything properly | 
  [releng] | 
            
  | 13:52 | 
  <hashar> | 
  making parsoid05 a Jenkins slave to replace parsoid04 | 
  [releng] | 
            
  | 13:24 | 
  <hashar> | 
  apt-get upgrade on parsoidcache02 and parsoid04 | 
  [releng] | 
            
  | 13:23 | 
  <hashar> | 
  updated labs/private on puppet master to fix a puppet dependency cycle with sudo-ldap | 
  [releng] | 
            
  | 13:19 | 
  <hashar> | 
  rebased puppetmaster repo | 
  [releng] | 
            
  | 12:52 | 
  <hashar> | 
  deleting the workspace for the beta-parsoid-update-eqiad jenkins job on deployment-parsoid04 . Some file belong to root which prevent the job from processing | 
  [releng] | 
            
  | 09:13 | 
  <hashar> | 
  enabled MediaWiki core 'structure' PHPUnit tests for all extensions.  Will require folks to fix their incorrect AutoLoader and  RessourceLoader entries. {{gerrit|180496}} {{bug|T78798}} | 
  [releng] | 
            
  
    | 
      
        2014-12-16
      
      §
     | 
  
    
  | 17:17 | 
  <bd808> | 
  git-sync-upstream runs cleanly on deployment-salt again! | 
  [releng] | 
            
  | 17:16 | 
  <bd808> | 
  removed cherry pick of Ib2a0401a7aa5632fb79a5b17c0d0cef8955cf990 (-2 by _joe_; replaced by Ibcad98a95413044fd6c5e9bd3c0a6fb486bd5fe9) | 
  [releng] | 
            
  | 17:15 | 
  <bd808> | 
  removed cherry pick of I3b6e37a2b6b9389c1a03bd572f422f898970c5b4 (modified in gerrit by bd808 and not repicked; merged) | 
  [releng] | 
            
  | 17:15 | 
  <bd808> | 
  removed cherry pick of I08c24578596506a1a8baedb7f4a42c2c78be295a (-2 by _joe_ in gerrit; replaced by Iba742c94aa3df7497fbff52a856d7ba16cf22cc7) | 
  [releng] | 
            
  | 17:13 | 
  <bd808> | 
  removed cherry pick of I6084f49e97c855286b86dbbd6ce8e80e94069492 (merged by Ori with a change) | 
  [releng] | 
            
  | 17:09 | 
  <bd808> | 
  trying to fix it without using important changes | 
  [releng] | 
            
  | 17:08 | 
  <bd808> | 
  deployment-salt:/var/lib/git/operations/puppet is a rebase hell of cherry-picks that don't apply | 
  [releng] | 
            
  | 16:52 | 
  <manybubbles> | 
  elasticsearch restart finished | 
  [releng] | 
            
  | 16:48 | 
  <mutante> | 
  deployment-db2 is down | 
  [releng] | 
            
  | 16:48 | 
  <manybubbles> | 
  restarting beta's elasticsearch servers to pick up a new version of a plugin.  won't interfere with current downtime. | 
  [releng] | 
            
  | 13:51 | 
  <hashar> | 
  deleting integration-slave1001 and recreating it. It is blocked on boot and we can't console on it https://phabricator.wikimedia.org/T76250 | 
  [releng] | 
            
  
    | 
      
        2014-12-13
      
      §
     | 
  
    
  | 18:51 | 
  <bd808> | 
  Running chmod -R g+s /data/project/upload7 on deploymnet-mediawiki02 | 
  [releng] | 
            
  | 18:25 | 
  <bd808> | 
  Running chmod -R u=rwX,g=rwX,o=rX /data/project/upload7 from deployment-mediawiki02 | 
  [releng] | 
            
  | 18:16 | 
  <bd808> | 
  chown done for /data/project/upload7 | 
  [releng] | 
            
  | 17:51 | 
  <bd808> | 
  Running chown -R apache:apache on /data/project/upload7 from deployment-mediawiki02 | 
  [releng] | 
            
  | 17:11 | 
  <bd808> | 
  Labs DNS seems to be flaking out badly and causing random scap and puppet failures | 
  [releng] | 
            
  | 17:10 | 
  <bd808> | 
  Many strange puppet and scap failures in beta that look to be related to DNS failures | 
  [releng] | 
            
  | 16:58 | 
  <bd808> | 
  restarted puppetmaster on deployment-salt | 
  [releng] | 
            
  | 16:31 | 
  <bd808> | 
  apache user renumbered on deployment-mediawiki03 | 
  [releng] | 
            
  | 16:23 | 
  <bd808> | 
  apache and hhvm restarted on beta app servers following apache user renumber | 
  [releng] | 
            
  | 16:09 | 
  <bd808> | 
  apache and hhvm stopped on beta app server tier. All requests expected to return 503 from varnish | 
  [releng] | 
            
  | 16:03 | 
  <bd808> | 
  Starting work on [[phab:T78076]] to renumber apache users in beta | 
  [releng] | 
            
  | 16:03 | 
  <bd808> | 
  Starting work on [[phab:T78076]] to renumber apache users in beta | 
  [releng] | 
            
  | 08:21 | 
  <YuviPanda|zzz> | 
  forcing puppet run on all deployment-prep hosts | 
  [releng] |