| 
      
        2014-07-29
      
      §
     | 
  
    
  | 19:14 | 
  <hashar> | 
  Removed all jobs from queue, restarted slave agent.  Update Jobs coming back | 
  [releng] | 
            
  | 19:09 | 
  <hashar> | 
  deployment-bastion jenkins slave is stuck.  Beta cluster is no more updating code :-// | 
  [releng] | 
            
  | 15:58 | 
  <godog> | 
  restarted hhvm on deploymnet-mediawiki01 | 
  [releng] | 
            
  | 15:52 | 
  <godog> | 
  restarted hhvm on deployment-mediawiki02 | 
  [releng] | 
            
  | 15:50 | 
  <godog> | 
  installed libevent-dbg on deployment-mediawiki02 to capture an hhvm backtrace | 
  [releng] | 
            
  | 15:17 | 
  <bd808> | 
  _joe_ restarting hhvm on deployment-mediawiki01 | 
  [releng] | 
            
  | 15:00 | 
  <bd808> | 
  Apache stuck with 65 children on both deployment-mediawiki servers | 
  [releng] | 
            
  | 10:37 | 
  <hashar> | 
  Restarted hhvm on mediawiki{01,02} | 
  [releng] | 
            
  
    | 
      
        2014-07-28
      
      §
     | 
  
    
  | 17:41 | 
  <bd808> | 
  Updated hhvm to latest 3.3-dev+20140728 build on deployment-mediawiki0[12] | 
  [releng] | 
            
  | 15:37 | 
  <manybubbles> | 
  rebuilding elasticsearch indexes to build a weighted all field we'll try to use to improve performance | 
  [releng] | 
            
  | 15:32 | 
  <bd808> | 
  Restarted hhvm on deployment-mediawiki0[12]. All apache children were stuck waiting for hhvm to respond. | 
  [releng] | 
            
  | 15:20 | 
  <bd808> | 
  Restarted apache on deployment-mediawiki02. 65 children and non-responsive to requests. (same as mediawiki01) | 
  [releng] | 
            
  | 15:18 | 
  <bd808> | 
  Restarted apache on deployment-mediawiki01. 65 children and non-responsive to requests. | 
  [releng] | 
            
  | 14:23 | 
  <manybubbles> | 
  or not - looks like I can't! | 
  [releng] | 
            
  | 14:22 | 
  <manybubbles> | 
  reubilding cirrus search indexes to pick up a speed up all field | 
  [releng] | 
            
  | 08:30 | 
  <hashar> | 
  restarted varnish on deployment-cache-bits01 . Hoping to clear bits cache | 
  [releng] | 
            
  
    | 
      
        2014-07-21
      
      §
     | 
  
    
  | 23:37 | 
  <ori> | 
  Switched over beta cluster app servers to HHVM | 
  [releng] | 
            
  | 21:27 | 
  <bd808> | 
  Killed update.php jobs; Antoine will give jobs a longer timeout | 
  [releng] | 
            
  | 21:23 | 
  <bd808> | 
  Running update.php for simplewiki in screen | 
  [releng] | 
            
  | 21:22 | 
  <bd808> | 
  Running update.php for hewiki in screen | 
  [releng] | 
            
  | 21:21 | 
  <bd808> | 
  Running update.php for eswiki in screen | 
  [releng] | 
            
  | 21:21 | 
  <bd808> | 
  Running update.php for cawiki in screen | 
  [releng] | 
            
  | 21:21 | 
  <bd808> | 
  Running update.php for commonswiki in screen | 
  [releng] | 
            
  | 21:18 | 
  <hashar> | 
  Restarting upd2log-mw on deployment-bastion.  There is a bunch of [python] <defunct> processes | 
  [releng] | 
            
  | 17:32 | 
  <bd808> | 
  Updated scap to 4871208 (+ cherry pick of I6a56b5e)  | 
  [releng] | 
            
  | 17:12 | 
  <bd808> | 
  Hotfix for scap ssh host key checking to fix jenkins scap job | 
  [releng] | 
            
  | 17:03 | 
  <bd808> | 
  Testing scap change I40a891b via cherry-pick | 
  [releng] | 
            
  | 10:25 | 
  <hashar> | 
  on bastion, fixed some puppet dependency to have nutcracker to start with the proper configuration {{gerrit|148043}} | 
  [releng] | 
            
  | 10:20 | 
  <hashar> | 
  upgrading packages on deployment-bastion | 
  [releng] | 
            
  | 10:19 | 
  <hashar> | 
  deleted /var/lib/apt/lists/lock on bastion. Was prevent apt-get update from running | 
  [releng] |