| 
      
        2012-08-03
      
      §
     | 
  
    
  | 09:49 | 
  <hashar> | 
  On extensions: git submodule foreach git checkout master | 
  [releng] | 
            
  | 09:13 | 
  <hashar> | 
  deleting deployment-bastion, it lacks a DNS entry {{bug|38846}} | 
  [releng] | 
            
  | 09:12 | 
  <hashar> | 
  running git gc --aggressive in  /home/wikipedia/common/php-master | 
  [releng] | 
            
  | 08:48 | 
  <hashar> | 
  making sure perms are correct in /data/project/apache/common-local/php-master : chmod -R g+w . ; chown mwdeploy:svn -R * .* | 
  [releng] | 
            
  | 08:45 | 
  <hashar> | 
  On -dbdump /etc/init.d/udp2log stop && /etc/init.d/udp2log-mw start  {{bug|38995}} | 
  [releng] | 
            
  | 08:35 | 
  <hashar> | 
  running puppet on Apache32 / 33  | 
  [releng] | 
            
  | 08:12 | 
  <hashar> | 
  dist-upgrading all instances to get the latest GlusterFS version (3.3.0) | 
  [releng] | 
            
  | 08:11 | 
  <hashar> | 
  Dist upgrading apache32 and 33 and rebooting them | 
  [releng] | 
            
  
    | 
      
        2012-07-27
      
      §
     | 
  
    
  | 22:50 | 
  <Platonides> | 
  Filesystem corruption signs in deployment-bastion, most debconf backends /usr/share/perl5/Debconf/FrontEnd are zeroed files. This explains some of the earlier apt-get problems, and maybe also bug 38747 (aka. magic tail -f fix) | 
  [releng] | 
            
  | 22:49 | 
  <Damianz> | 
  Thumbnails are currently broken - thumbs folder seems empty, images are there as expected - beta quirk or need rebuilding? | 
  [releng] | 
            
  | 22:22 | 
  <Damianz> | 
  DocumentRoot got wiped, broken all the sites - fixed the broken symlink on bastion for /data/project/apache to /usr/local/apache. Also fixed common-local to common symlink. | 
  [releng] | 
            
  | 21:56 | 
  <Platonides> | 
  apt-get removed wikimedia-task-appserver in deployment-bastion :( | 
  [releng] | 
            
  | 21:42 | 
  <Platonides> | 
  removed generated captcha files | 
  [releng] | 
            
  | 21:33 | 
  <Platonides> | 
  installed in deployment-bastion the packages joe, python-imaging and wamerican | 
  [releng] | 
            
  | 20:29 | 
  <labs-logs-bottie> | 
  j: deployment-bastion: removing all deployment-nfs-memc entries from /etc/fstab | 
  [releng] | 
            
  | 13:14 | 
  <hashar> | 
  restarted job runner on jobrunner06 | 
  [releng] | 
            
  | 13:11 | 
  <hashar> | 
  removing all deployment-nfs-memc entries from /etc/fstab | 
  [releng] | 
            
  | 09:51 | 
  <hashar> | 
  {{bug|38749}} jobrunner06 : removed /usr/local/apache and /mnt/upload6 empty dir. Downgraded PHP manually. Rerunning puppet. | 
  [releng] | 
            
  | 08:46 | 
  <hashar> | 
  migrating jobrunner06 to use the /data/project for uploads | 
  [releng] | 
            
  
    | 
      
        2012-07-26
      
      §
     | 
  
    
  | 19:27 | 
  <labs-logs-bottie> | 
  j: delete broken deployment-video01 instance | 
  [releng] | 
            
  | 18:42 | 
  <labs-logs-bottie> | 
  j: new instance deployment-video02 with videoscaler with access to gluster instead of nfs | 
  [releng] | 
            
  | 16:15 | 
  <hashar> | 
  applying nfs::apache::labs on -dbdump to get /usr/local/apache from /data/project | 
  [releng] | 
            
  | 16:12 | 
  <hashar> | 
  -dbdump umounted deployment-nfs-memc:/mnt/export/apache on /usr/local/apache | 
  [releng] | 
            
  | 13:56 | 
  <hashar> | 
  archiving -nfs-memc:/mnt/export : root@deployment-nfs-memc:/mnt# mv export /data/project/deployment-nfs-memc_mnt-export_backup  | 
  [releng] | 
            
  | 13:46 | 
  <hashar> | 
  seems to work fine with /data/project now.  | 
  [releng] | 
            
  | 13:46 | 
  <hashar> | 
  rsync finished for both apache and upload6. Remounting and restarting apaches | 
  [releng] | 
            
  | 13:35 | 
  <hashar> | 
  rsync from nfs-memc:/mnt/export/upload6 to /data/project/upload6 completed. YEAHHH | 
  [releng] | 
            
  | 13:32 | 
  <hashar> | 
  on -dbdump, unmounted /mnt/upload (from nfs-memc). Please use the /mnt/upload6 -> /data/project/upload6 | 
  [releng] | 
            
  | 13:29 | 
  <hashar> | 
  root@deployment-nfs-memc:/mnt/export# rsync -a --progress --delete --inplace /mnt/export/apache /data/project | 
  [releng] | 
            
  | 13:22 | 
  <hashar> | 
  applying nfs::upload::labs  to -dbdump so it uses /data/project/upload6 at  /mnt/upload6 | 
  [releng] | 
            
  | 13:15 | 
  <hashar> | 
  manually umount /mnt/upload6 on apaches  | 
  [releng] | 
            
  | 13:14 | 
  <hashar> | 
  stopping apache backends | 
  [releng] | 
            
  | 13:10 | 
  <hashar> | 
  Manually running puppet for {{gerrit|15545}} which should fix  {{bug|38084}} uses /data/project instead of NFS instance | 
  [releng] | 
            
  | 09:35 | 
  <hashar> | 
  Regenerating captcha with the new shared key fixed {{bug|38699}} | 
  [releng] | 
            
  | 09:26 | 
  <hashar> | 
  Deleted all captchas in /mnt/upload6/private/captcha and regenerate them using 1000 and dirs=3 | 
  [releng] | 
            
  | 09:22 | 
  <hashar> | 
  Set $wmgCaptchaSecret in the local file common/wmf-config/PrivateSettings.php and used that value in captcha.py | 
  [releng] |