| 2015-03-27
      
      § | 
    
  | 23:28 | <bd808> | applied beta::autoupdater directly to deployment-bastion via wikitech interface | [releng] | 
            
  | 23:21 | <bd808> | Duplicate declaration: Git::Clone[operations/mediawiki-config] is already declared in file /etc/puppet/modules/beta/manifests/autoupdater.pp:46; cannot redeclare at /etc/puppet/modules/scap/manifests/master.pp:22 | [releng] | 
            
  | 23:01 | <bd808> | restarted puppetmaster | [releng] | 
            
  | 22:52 | <hashar> | integration: jzerebecki addition and sudo policy  tracked for history purpose as {{bug|T94280}} | [releng] | 
            
  | 22:52 | <bd808> | chown -R l10nupdate:wikidev /srv/mediawiki-staging/php-master/cache/l10n | [releng] | 
            
  | 22:44 | <bd808> | deployment-bastion: chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/ | [releng] | 
            
  | 22:41 | <bd808> | forcing puppet run on deployment-bastion | [releng] | 
            
  | 22:41 | <bd808> | cherry-picked https://gerrit.wikimedia.org/r/#/c/200248/ and https://gerrit.wikimedia.org/r/#/c/199988/ | [releng] | 
            
  | 22:40 | <hashar> | integration: created sudo policy allowing members to run any command as jenkins-deploy on all hosts. | [releng] | 
            
  | 22:40 | <hashar> | added jzerebecki to the integration labs project as a normal member | [releng] | 
            
  | 22:34 | <hashar> | integration-slave1001 rm -fR mwext-Wikibase-repo-api-tests/src/vendor | [releng] | 
            
  | 21:13 | <greg-g> | things be better | [releng] | 
            
  | 20:56 | <greg-g> | Beta Cluster is down, known | [releng] | 
            
  | 18:50 | <marxarelli> | running `jenkins-jobs update` to update 'browsertests-UploadWizard-*' with Id33ffde07f0c15e153d52388cf130be4c59b4559 | [releng] | 
            
  | 17:50 | <legoktm> | deleted core dumps from integration-slave1002 | [releng] | 
            
  | 17:48 | <legoktm> | marked integration-slave1002 as offline, /var filled up | [releng] | 
            
  | 05:42 | <legoktm> | marked integration-slave1001 as offline due to https://phabricator.wikimedia.org/T94138 | [releng] | 
            
  
    | 2015-03-26
      
      § | 
    
  | 23:47 | <legoktm> | deploying https://gerrit.wikimedia.org/r/200069 | [releng] | 
            
  | 19:22 | <bd808> | Manually added missing !log entries from 2015-03-25 from my bouncer logs | [releng] | 
            
  | 17:14 | <greg-g> | jobs appear to be processing according to zuul, the Jenkins UI just takes forever to load, apparently | [releng] | 
            
  | 17:12 | <greg-g> | "Please wait while Jenkins is getting ready to work" | [releng] | 
            
  | 17:08 | <greg-g> | 0:07 <      robh> kill -9 and restarted per instrucitons | [releng] | 
            
  | 16:53 | <greg-g> | Still.... "Please wait while Jenkins is restarting..." | [releng] | 
            
  | 16:49 | <greg-g> | "Please wait while Jenkins is restarting..." | [releng] | 
            
  | 16:39 | <greg-g> | going to do a safe-restart of Jenkins https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Restart_all_of_Jenkins | [releng] | 
            
  | 16:38 | <greg-g> | nothing executing on deployment-bastion, that is | [releng] | 
            
  | 16:38 | <greg-g> | same, nothing executing | [releng] | 
            
  | 16:37 | <greg-g> | did that checklist once, jobs still not executing, doing again | [releng] | 
            
  | 16:32 | <greg-g> | I'll start going through the checklist at https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update | [releng] | 
            
  | 16:30 | <hashar> | deadlock on deployment-bastion slave. Someone need to restart Jenkins :( | [releng] | 
            
  | 13:25 | <hashar> | yamllint job fixed by altering the label https://gerrit.wikimedia.org/r/#/c/199876/ | [releng] | 
            
  | 13:17 | <hashar> | Changes blocked because there is nothing able to run yamllint  ( zuul-gearman.py status|grep build:yamllint  ,  shows 8 jobs pending and no worker available) | [releng] | 
            
  
    | 2015-03-24
      
      § | 
    
  | 23:32 | <Krinkle> | Force restart Zuul | [releng] | 
            
  | 22:25 | <hashar> | marked gallium and lanthanum slaves as temp offline, then back. Seems to have cleared some Jenkins internal state and resumed the build | [releng] | 
            
  | 21:55 | <bd808> | Ran trebuchet for scap to keep cherry-pick of I01b24765ce26cf48d9b9381a476c3bcf39db7ab8 on top of active branch; puppet was forcing back to prior trebuchet sync tag | [releng] | 
            
  | 21:42 | <hashar> | Reconfigured [https://integration.wikimedia.org/ci/view/Beta/job/mediawiki-core-code-coverage/ mediawiki-core-code-coverage] | [releng] | 
            
  | 21:22 | <hashar> | Zuul gate is deadlocked for up to half an hour due to change being force merged :( | [releng] | 
            
  | 21:15 | <hashar> | beta: deleted untracked file /srv/mediawiki-staging/php-master/extensions/.gitignore . That fixed the Jenkins job https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ | [releng] | 
            
  | 20:31 | <twentyafterfour> | sudo ln -s /srv/l10nupdate/ /var/lib/ | [releng] | 
            
  | 20:31 | <twentyafterfour> | sudo mv /var/lib/l10nupdate/ /srv/ | [releng] | 
            
  | 20:28 | <bd808> | deployment-bastion -- rm -r pacct.1.gz pacct.2.gz pacct.3.gz pacct.4.gz pacct.5.gz pacct.6.gz | [releng] | 
            
  | 20:24 | <bd808> | Deleted junk in deployment-bastion:/tmp | [releng] | 
            
  | 18:57 | <legoktm> | deploying https://gerrit.wikimedia.org/r/199305 | [releng] | 
            
  | 18:25 | <legoktm> | deploying https://gerrit.wikimedia.org/r/199216 | [releng] | 
            
  | 17:06 | <legoktm> | deploying https://gerrit.wikimedia.org/r/199273 | [releng] | 
            
  | 11:23 | <hashar> | beta-scap-eqiad keeps regenerating l10n cache https://phabricator.wikimedia.org/T93737 | [releng] | 
            
  | 08:35 | <hashar> | restarting Jenkins for some plugins upgrades | [releng] |