| 
      
        2012-05-23
      
      §
     | 
  
    
  | 19:54 | 
  <hashar> | 
  rebooting apache20 following installation of imagescaler puppet class | 
  [releng] | 
            
  | 19:49 | 
  <hashar> | 
  rebooting apache23 following installation of imagescaler puppet class | 
  [releng] | 
            
  | 19:46 | 
  <hashar> | 
  rebooting apache22 following installation of imagescaler puppet class | 
  [releng] | 
            
  | 19:15 | 
  <hashar> | 
  rebooting apache21 following installation of imagescaler puppet class | 
  [releng] | 
            
  | 18:48 | 
  <hashar> | 
  Adding puppet class 'imagescaler' on all deployment-apacheXX instances in an attempt to fix thumbnails | 
  [releng] | 
            
  | 18:36 | 
  <labs-logs-bottie> | 
  hashar: relocalisation cache done 367/367 languages rebuilt | 
  [releng] | 
            
  | 18:28 | 
  <labs-logs-bottie> | 
  hashar: running `mwscript rebuildLocalisationCache.php --wiki=aawiki` for {{bug|36806}} | 
  [releng] | 
            
  | 16:22 | 
  <labs-logs-bottie> | 
  hashar: delete all 3 webVideoTranscode jobs from enwiki database | 
  [releng] | 
            
  | 16:10 | 
  <hashar> | 
  rebooted jobrunner03 to check everything works fine there | 
  [releng] | 
            
  | 15:41 | 
  <hashar> | 
  deleting jobrunner01, it is crashed beyond repair. Will create a new one named jobrunner03 | 
  [releng] | 
            
  | 15:27 | 
  <hashar> | 
  rebooting jobrunner01 to see how it goes | 
  [releng] | 
            
  | 14:56 | 
  <hashar> | 
  stopped job runner on jobrunner01,  amounted /mnt/upload6 and /mnt/ | 
  [releng] | 
            
  | 14:37 | 
  <hashar> | 
  running puppet on job runner to check change 8584 & 8585 worked | 
  [releng] | 
            
  
    | 
      
        2012-05-22
      
      §
     | 
  
    
  | 13:49 | 
  <hashar> | 
  Deleting jobrunner03 and 04, not going to need them afterall | 
  [releng] | 
            
  | 13:24 | 
  <hashar> | 
  deleting refreshLinks2 jobs from enwiki database | 
  [releng] | 
            
  | 12:52 | 
  <hashar> | 
  deleting deployment-jobrunner{3,4} installation failed I got permission denied. Will recreate them using same hostname | 
  [releng] | 
            
  | 11:55 | 
  <hashar> | 
  create two more job runner instances | 
  [releng] | 
            
  | 10:09 | 
  <hashar> | 
  Remove deployment-webs instance which was meant to emulate the HTTPS access. Hacky and low priority for now, we will need to setup a nginx proxy one day to properly replicate the production infrastructure. | 
  [releng] | 
            
  | 09:39 | 
  <labs-logs-bottie> | 
  hashar: rebooting jobrunner02 just to be sure it is properly loaded up | 
  [releng] | 
            
  | 09:30 | 
  <labs-logs-bottie> | 
  hashar: jobrunner logs are available in /home/wikipedia/logs/runJobs.log now | 
  [releng] | 
            
  | 09:25 | 
  <hashar> | 
  Fixed udp2log not able to add new log files in /home/wikipedia/log , that dir need to be writable by udp2log user!  See https://gerrit.wikimedia.org/r/8442 | https://bugzilla.wikimedia.org/37014 | 
  [releng] | 
            
  | 08:54 | 
  <hashar> | 
  purged all logs from /home/wikipedia/logs/archive/ just to be safe | 
  [releng] | 
            
  | 08:41 | 
  <hashar> | 
  restarted upd2log on -feed (again) | 
  [releng] | 
            
  | 08:23 | 
  <hashar> | 
  started job loop on deployment-job-runner02 | 
  [releng] | 
            
  | 07:49 | 
  <hashar> | 
  installing jobrunner2 | 
  [releng] | 
            
  | 05:11 | 
  <hashar> | 
  creating a second job runner instance deployment-jobrunner02 . Will apply puppet classes later on. | 
  [releng] | 
            
  | 03:47 | 
  <labs-logs-bottie> | 
  hashar: (Bug 36870) deleting deployment-web{,3,4,5} | 
  [releng] | 
            
  
    | 
      
        2012-05-21
      
      §
     | 
  
    
  | 21:11 | 
  <hashar> | 
  deployment-nfs-memc : fix user right for upload6 : chown apache /mnt/export/upload6 | 
  [releng] | 
            
  | 21:11 | 
  <hashar> | 
  On deployment-nfs-memc : added apache (uid 48) entry in /etc/passwd | 
  [releng] | 
            
  | 21:11 | 
  <hashar> | 
  Adding Faidon and Platonides to default CC list of "depoyment prep (beta)" component | 
  [releng] | 
            
  | 21:11 | 
  <hashar> | 
  In Bugzilla, I have removed Petr Bena as a default assignee of bugs opened for "deployment-prep (beta)" component. Default is now "Nobody", Petr is on CC. That will makes bug triage a bit easier. | 
  [releng] | 
            
  | 08:58 | 
  <hashar> | 
  rerebooting deployment-feed | 
  [releng] | 
            
  | 08:53 | 
  <hashar> | 
  Looks like -feed is dead : EXT3-fs: INFO: recovery required on readonly filesystem. | 
  [releng] | 
            
  | 08:49 | 
  <hashar> | 
  rebooting deployment-feed  | 
  [releng] | 
            
  | 08:22 | 
  <hashar> | 
  installing iotop on deployment-nfs-memc | 
  [releng] |