| 2012-05-23
      
      § | 
    
  | 19:54 | <hashar> | rebooting apache20 following installation of imagescaler puppet class | [releng] | 
            
  | 19:49 | <hashar> | rebooting apache23 following installation of imagescaler puppet class | [releng] | 
            
  | 19:46 | <hashar> | rebooting apache22 following installation of imagescaler puppet class | [releng] | 
            
  | 19:15 | <hashar> | rebooting apache21 following installation of imagescaler puppet class | [releng] | 
            
  | 18:48 | <hashar> | Adding puppet class 'imagescaler' on all deployment-apacheXX instances in an attempt to fix thumbnails | [releng] | 
            
  | 18:36 | <labs-logs-bottie> | hashar: relocalisation cache done 367/367 languages rebuilt | [releng] | 
            
  | 18:28 | <labs-logs-bottie> | hashar: running `mwscript rebuildLocalisationCache.php --wiki=aawiki` for {{bug|36806}} | [releng] | 
            
  | 16:22 | <labs-logs-bottie> | hashar: delete all 3 webVideoTranscode jobs from enwiki database | [releng] | 
            
  | 16:10 | <hashar> | rebooted jobrunner03 to check everything works fine there | [releng] | 
            
  | 15:41 | <hashar> | deleting jobrunner01, it is crashed beyond repair. Will create a new one named jobrunner03 | [releng] | 
            
  | 15:27 | <hashar> | rebooting jobrunner01 to see how it goes | [releng] | 
            
  | 14:56 | <hashar> | stopped job runner on jobrunner01,  amounted /mnt/upload6 and /mnt/ | [releng] | 
            
  | 14:37 | <hashar> | running puppet on job runner to check change 8584 & 8585 worked | [releng] | 
            
  
    | 2012-05-22
      
      § | 
    
  | 13:49 | <hashar> | Deleting jobrunner03 and 04, not going to need them afterall | [releng] | 
            
  | 13:24 | <hashar> | deleting refreshLinks2 jobs from enwiki database | [releng] | 
            
  | 12:52 | <hashar> | deleting deployment-jobrunner{3,4} installation failed I got permission denied. Will recreate them using same hostname | [releng] | 
            
  | 11:55 | <hashar> | create two more job runner instances | [releng] | 
            
  | 10:09 | <hashar> | Remove deployment-webs instance which was meant to emulate the HTTPS access. Hacky and low priority for now, we will need to setup a nginx proxy one day to properly replicate the production infrastructure. | [releng] | 
            
  | 09:39 | <labs-logs-bottie> | hashar: rebooting jobrunner02 just to be sure it is properly loaded up | [releng] | 
            
  | 09:30 | <labs-logs-bottie> | hashar: jobrunner logs are available in /home/wikipedia/logs/runJobs.log now | [releng] | 
            
  | 09:25 | <hashar> | Fixed udp2log not able to add new log files in /home/wikipedia/log , that dir need to be writable by udp2log user!  See https://gerrit.wikimedia.org/r/8442 | https://bugzilla.wikimedia.org/37014 | [releng] | 
            
  | 08:54 | <hashar> | purged all logs from /home/wikipedia/logs/archive/ just to be safe | [releng] | 
            
  | 08:41 | <hashar> | restarted upd2log on -feed (again) | [releng] | 
            
  | 08:23 | <hashar> | started job loop on deployment-job-runner02 | [releng] | 
            
  | 07:49 | <hashar> | installing jobrunner2 | [releng] | 
            
  | 05:11 | <hashar> | creating a second job runner instance deployment-jobrunner02 . Will apply puppet classes later on. | [releng] | 
            
  | 03:47 | <labs-logs-bottie> | hashar: (Bug 36870) deleting deployment-web{,3,4,5} | [releng] | 
            
  
    | 2012-05-21
      
      § | 
    
  | 21:11 | <hashar> | deployment-nfs-memc : fix user right for upload6 : chown apache /mnt/export/upload6 | [releng] | 
            
  | 21:11 | <hashar> | On deployment-nfs-memc : added apache (uid 48) entry in /etc/passwd | [releng] | 
            
  | 21:11 | <hashar> | Adding Faidon and Platonides to default CC list of "depoyment prep (beta)" component | [releng] | 
            
  | 21:11 | <hashar> | In Bugzilla, I have removed Petr Bena as a default assignee of bugs opened for "deployment-prep (beta)" component. Default is now "Nobody", Petr is on CC. That will makes bug triage a bit easier. | [releng] | 
            
  | 08:58 | <hashar> | rerebooting deployment-feed | [releng] | 
            
  | 08:53 | <hashar> | Looks like -feed is dead : EXT3-fs: INFO: recovery required on readonly filesystem. | [releng] | 
            
  | 08:49 | <hashar> | rebooting deployment-feed | [releng] | 
            
  | 08:22 | <hashar> | installing iotop on deployment-nfs-memc | [releng] |