| 
      
        2015-03-26
      
      §
     | 
  
    
  | 23:47 | 
  <legoktm> | 
  deploying https://gerrit.wikimedia.org/r/200069 | 
  [releng] | 
            
  | 19:22 | 
  <bd808> | 
  Manually added missing !log entries from 2015-03-25 from my bouncer logs | 
  [releng] | 
            
  | 17:14 | 
  <greg-g> | 
  jobs appear to be processing according to zuul, the Jenkins UI just takes forever to load, apparently | 
  [releng] | 
            
  | 17:12 | 
  <greg-g> | 
  "Please wait while Jenkins is getting ready to work" | 
  [releng] | 
            
  | 17:08 | 
  <greg-g> | 
  0:07 <      robh> kill -9 and restarted per instrucitons | 
  [releng] | 
            
  | 16:53 | 
  <greg-g> | 
  Still.... "Please wait while Jenkins is restarting..." | 
  [releng] | 
            
  | 16:49 | 
  <greg-g> | 
  "Please wait while Jenkins is restarting..." | 
  [releng] | 
            
  | 16:39 | 
  <greg-g> | 
  going to do a safe-restart of Jenkins https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Restart_all_of_Jenkins | 
  [releng] | 
            
  | 16:38 | 
  <greg-g> | 
  nothing executing on deployment-bastion, that is | 
  [releng] | 
            
  | 16:38 | 
  <greg-g> | 
  same, nothing executing | 
  [releng] | 
            
  | 16:37 | 
  <greg-g> | 
  did that checklist once, jobs still not executing, doing again | 
  [releng] | 
            
  | 16:32 | 
  <greg-g> | 
  I'll start going through the checklist at https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update | 
  [releng] | 
            
  | 16:30 | 
  <hashar> | 
  deadlock on deployment-bastion slave. Someone need to restart Jenkins :( | 
  [releng] | 
            
  | 13:25 | 
  <hashar> | 
  yamllint job fixed by altering the label https://gerrit.wikimedia.org/r/#/c/199876/ | 
  [releng] | 
            
  | 13:17 | 
  <hashar> | 
  Changes blocked because there is nothing able to run yamllint  ( zuul-gearman.py status|grep build:yamllint  ,  shows 8 jobs pending and no worker available) | 
  [releng] | 
            
  
    | 
      
        2015-03-24
      
      §
     | 
  
    
  | 23:32 | 
  <Krinkle> | 
  Force restart Zuul | 
  [releng] | 
            
  | 22:25 | 
  <hashar> | 
  marked gallium and lanthanum slaves as temp offline, then back. Seems to have cleared some Jenkins internal state and resumed the build | 
  [releng] | 
            
  | 21:55 | 
  <bd808> | 
  Ran trebuchet for scap to keep cherry-pick of I01b24765ce26cf48d9b9381a476c3bcf39db7ab8 on top of active branch; puppet was forcing back to prior trebuchet sync tag | 
  [releng] | 
            
  | 21:42 | 
  <hashar> | 
  Reconfigured [https://integration.wikimedia.org/ci/view/Beta/job/mediawiki-core-code-coverage/ mediawiki-core-code-coverage] | 
  [releng] | 
            
  | 21:22 | 
  <hashar> | 
  Zuul gate is deadlocked for up to half an hour due to change being force merged :( | 
  [releng] | 
            
  | 21:15 | 
  <hashar> | 
  beta: deleted untracked file /srv/mediawiki-staging/php-master/extensions/.gitignore . That fixed the Jenkins job https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ | 
  [releng] | 
            
  | 20:31 | 
  <twentyafterfour> | 
  sudo ln -s /srv/l10nupdate/ /var/lib/ | 
  [releng] | 
            
  | 20:31 | 
  <twentyafterfour> | 
  sudo mv /var/lib/l10nupdate/ /srv/ | 
  [releng] | 
            
  | 20:28 | 
  <bd808> | 
  deployment-bastion -- rm -r pacct.1.gz pacct.2.gz pacct.3.gz pacct.4.gz pacct.5.gz pacct.6.gz | 
  [releng] | 
            
  | 20:24 | 
  <bd808> | 
  Deleted junk in deployment-bastion:/tmp | 
  [releng] | 
            
  | 18:57 | 
  <legoktm> | 
  deploying https://gerrit.wikimedia.org/r/199305 | 
  [releng] | 
            
  | 18:25 | 
  <legoktm> | 
  deploying https://gerrit.wikimedia.org/r/199216 | 
  [releng] | 
            
  | 17:06 | 
  <legoktm> | 
  deploying https://gerrit.wikimedia.org/r/199273 | 
  [releng] | 
            
  | 11:23 | 
  <hashar> | 
  beta-scap-eqiad keeps regenerating l10n cache https://phabricator.wikimedia.org/T93737   | 
  [releng] | 
            
  | 08:35 | 
  <hashar> | 
  restarting Jenkins for some plugins upgrades | 
  [releng] | 
            
  | 08:07 | 
  <legoktm> | 
  deployed https://gerrit.wikimedia.org/r/199190 | 
  [releng] | 
            
  | 07:21 | 
  <legoktm> | 
  deploying https://gerrit.wikimedia.org/r/199205 | 
  [releng] | 
            
  | 07:17 | 
  <legoktm> | 
  deploying https://gerrit.wikimedia.org/r/199204 | 
  [releng] | 
            
  | 07:08 | 
  <legoktm> | 
  deploying https://gerrit.wikimedia.org/r/199201 | 
  [releng] | 
            
  | 06:46 | 
  <legoktm> | 
  freed ~6G on lanthanum by deleting mediawiki-extensions-zend* worksapces | 
  [releng] | 
            
  | 05:04 | 
  <legoktm> | 
  deleting workspaces of jobs that no longer exist in jjb on lathanum | 
  [releng] | 
            
  | 04:11 | 
  <legoktm> | 
  deploying https://gerrit.wikimedia.org/r/198792 | 
  [releng] | 
            
  | 03:14 | 
  <Krinkle> | 
  Deleting old job workspaces on gallium not touched since 2013 | 
  [releng] | 
            
  | 02:42 | 
  <Krinkle> | 
  Restarting Zuul, wikimedia-fundraising-civicrm is stuck as of 46min ago waiting for something already merged | 
  [releng] | 
            
  | 02:32 | 
  <legoktm> | 
  toggling gearman off/on in jenkins | 
  [releng] | 
            
  | 01:47 | 
  <twentyafterfour> | 
  deployed scap/scap-sync-20150324-014257 to beta cluster | 
  [releng] | 
            
  | 00:23 | 
  <Krinkle> | 
  Restarted Zuul | 
  [releng] |