| 
      
        2014-08-15
      
      §
     | 
  
    
  | 21:57 | 
  <legoktm> | 
  set $wgVERPsecret in PrivateSettings.php | 
  [releng] | 
            
  | 21:42 | 
  <hashSpeleology> | 
  Beta cluster database updates are broken due to CentralNotice. Fix up is {{gerrit|154231}} | 
  [releng] | 
            
  | 20:57 | 
  <hashSpeleology> | 
  deployment-rsync01 : deleting /usr/local/apache/common-local content.  Then ln -s /srv/common-local /usr/local/apache/common-local  as set by beta::common which is not applied on that host for some reason.  {{bug|69590}} | 
  [releng] | 
            
  | 20:55 | 
  <hashSpeleology> | 
  puppet administratively disabled on mediawiki02 . Assuming some work in progress on that host. Leaving it untouched | 
  [releng] | 
            
  | 20:54 | 
  <hashSpeleology> | 
  puppet is proceeding on mediawiki01 | 
  [releng] | 
            
  | 20:52 | 
  <hashSpeleology> | 
  attempting to unbreak mediawiki code update {{bug|69590}} by cherry picking {{gerrit|154329}} | 
  [releng] | 
            
  | 20:39 | 
  <hashSpeleology> | 
  in case it is not in SAL.  MediaWiki is no more synced to app server {{bug|69590}} | 
  [releng] | 
            
  | 20:20 | 
  <hashSpeleology> | 
  rebooting mediawiki01  , /var refuses to clear out and stick at 100% usage | 
  [releng] | 
            
  | 20:16 | 
  <hashSpeleology> | 
  cleaning up /var/log on deployment-mediawiki02 | 
  [releng] | 
            
  | 20:14 | 
  <hashSpeleology> | 
  on deployment-mediawiki01 deleting /var/log/apache2/access.log.1 | 
  [releng] | 
            
  | 20:13 | 
  <hashSpeleology> | 
  on deployment-mediawiki01 deleting /var/log/apache2/debug.log.1 | 
  [releng] | 
            
  | 20:13 | 
  <hashSpeleology> | 
  bunch of instances have a full /var/log :-/ | 
  [releng] | 
            
  | 11:37 | 
  <ori> | 
  deployment-cache-bits01 unresponsive; console shows OOMs: https://dpaste.de/LDRi/raw . rebooting | 
  [releng] | 
            
  | 03:20 | 
  <jeremyb> | 
  02:46:37 UTC <ebernhardson> !log beta /dev/vda1 full. moved /srv-old to /mnt/srv-old and freed up 2.1G | 
  [releng] |