| 2012-08-03
      
      § | 
    
  | 09:53 | <hashar> | Updating mediawiki core to latest master: Updating 0c1471c..d47c1e9 (fast forwarded) | [releng] | 
            
  | 09:49 | <hashar> | On extensions: git submodule foreach git checkout master | [releng] | 
            
  | 09:13 | <hashar> | deleting deployment-bastion, it lacks a DNS entry {{bug|38846}} | [releng] | 
            
  | 09:12 | <hashar> | running git gc --aggressive in  /home/wikipedia/common/php-master | [releng] | 
            
  | 08:48 | <hashar> | making sure perms are correct in /data/project/apache/common-local/php-master : chmod -R g+w . ; chown mwdeploy:svn -R * .* | [releng] | 
            
  | 08:45 | <hashar> | On -dbdump /etc/init.d/udp2log stop && /etc/init.d/udp2log-mw start  {{bug|38995}} | [releng] | 
            
  | 08:35 | <hashar> | running puppet on Apache32 / 33 | [releng] | 
            
  | 08:12 | <hashar> | dist-upgrading all instances to get the latest GlusterFS version (3.3.0) | [releng] | 
            
  | 08:11 | <hashar> | Dist upgrading apache32 and 33 and rebooting them | [releng] | 
            
  
    | 2012-07-27
      
      § | 
    
  | 22:50 | <Platonides> | Filesystem corruption signs in deployment-bastion, most debconf backends /usr/share/perl5/Debconf/FrontEnd are zeroed files. This explains some of the earlier apt-get problems, and maybe also bug 38747 (aka. magic tail -f fix) | [releng] | 
            
  | 22:49 | <Damianz> | Thumbnails are currently broken - thumbs folder seems empty, images are there as expected - beta quirk or need rebuilding? | [releng] | 
            
  | 22:22 | <Damianz> | DocumentRoot got wiped, broken all the sites - fixed the broken symlink on bastion for /data/project/apache to /usr/local/apache. Also fixed common-local to common symlink. | [releng] | 
            
  | 21:56 | <Platonides> | apt-get removed wikimedia-task-appserver in deployment-bastion :( | [releng] | 
            
  | 21:42 | <Platonides> | removed generated captcha files | [releng] | 
            
  | 21:33 | <Platonides> | installed in deployment-bastion the packages joe, python-imaging and wamerican | [releng] | 
            
  | 20:29 | <labs-logs-bottie> | j: deployment-bastion: removing all deployment-nfs-memc entries from /etc/fstab | [releng] | 
            
  | 13:14 | <hashar> | restarted job runner on jobrunner06 | [releng] | 
            
  | 13:11 | <hashar> | removing all deployment-nfs-memc entries from /etc/fstab | [releng] | 
            
  | 09:51 | <hashar> | {{bug|38749}} jobrunner06 : removed /usr/local/apache and /mnt/upload6 empty dir. Downgraded PHP manually. Rerunning puppet. | [releng] | 
            
  | 08:46 | <hashar> | migrating jobrunner06 to use the /data/project for uploads | [releng] | 
            
  
    | 2012-07-26
      
      § | 
    
  | 19:27 | <labs-logs-bottie> | j: delete broken deployment-video01 instance | [releng] | 
            
  | 18:42 | <labs-logs-bottie> | j: new instance deployment-video02 with videoscaler with access to gluster instead of nfs | [releng] | 
            
  | 16:15 | <hashar> | applying nfs::apache::labs on -dbdump to get /usr/local/apache from /data/project | [releng] | 
            
  | 16:12 | <hashar> | -dbdump umounted deployment-nfs-memc:/mnt/export/apache on /usr/local/apache | [releng] | 
            
  | 13:56 | <hashar> | archiving -nfs-memc:/mnt/export : root@deployment-nfs-memc:/mnt# mv export /data/project/deployment-nfs-memc_mnt-export_backup | [releng] | 
            
  | 13:46 | <hashar> | seems to work fine with /data/project now. | [releng] | 
            
  | 13:46 | <hashar> | rsync finished for both apache and upload6. Remounting and restarting apaches | [releng] | 
            
  | 13:35 | <hashar> | rsync from nfs-memc:/mnt/export/upload6 to /data/project/upload6 completed. YEAHHH | [releng] | 
            
  | 13:32 | <hashar> | on -dbdump, unmounted /mnt/upload (from nfs-memc). Please use the /mnt/upload6 -> /data/project/upload6 | [releng] | 
            
  | 13:29 | <hashar> | root@deployment-nfs-memc:/mnt/export# rsync -a --progress --delete --inplace /mnt/export/apache /data/project | [releng] | 
            
  | 13:22 | <hashar> | applying nfs::upload::labs  to -dbdump so it uses /data/project/upload6 at  /mnt/upload6 | [releng] | 
            
  | 13:15 | <hashar> | manually umount /mnt/upload6 on apaches | [releng] | 
            
  | 13:14 | <hashar> | stopping apache backends | [releng] | 
            
  | 13:10 | <hashar> | Manually running puppet for {{gerrit|15545}} which should fix  {{bug|38084}} uses /data/project instead of NFS instance | [releng] | 
            
  | 09:35 | <hashar> | Regenerating captcha with the new shared key fixed {{bug|38699}} | [releng] | 
            
  | 09:26 | <hashar> | Deleted all captchas in /mnt/upload6/private/captcha and regenerate them using 1000 and dirs=3 | [releng] | 
            
  | 09:22 | <hashar> | Set $wmgCaptchaSecret in the local file common/wmf-config/PrivateSettings.php and used that value in captcha.py | [releng] |