| 2012-07-18
      
      § | 
    
  | 14:38 | <hashar> | New / rebooted instances are no more accessible : {{bug|38473}} - instances can not boot / reboot anymore | [releng] | 
            
  | 13:46 | <hashar> | deleting upload01 (screwed somehow) | [releng] | 
            
  | 13:46 | <hashar> | creating deployment-cache-upload03 to replace upload01 | [releng] | 
            
  | 13:44 | <hashar> | deployment-cache-upload01 seems screwed : waiting for metadata service at http://169.254.169.254/2009-04-04/meta-data/instance-id . Failed DHCP acquisition ? => rebooting | [releng] | 
            
  | 13:08 | <hashar> | deployment-cache-upload01 : running apt-get upgrade / dist-upgrade and rebooting | [releng] | 
            
  | 10:22 | <hashar> | copying apache dir to /data/project . Run as root@deployment-nfs-memc in a screen  session | [releng] | 
            
  | 09:02 | <hashar> | adding nfs::apache::labs and nfs::upload::labs to deployment-integration | [releng] | 
            
  | 08:59 | <hashar> | Applying {{gerrit|15545}} to deployment-integration | [releng] | 
            
  | 08:26 | <hashar> | Created deployment-integration to be used as a puppetmaster::self host | [releng] | 
            
  
    | 2012-07-17
      
      § | 
    
  | 21:26 | <Platonides> | installed python-imaging and wamerican on deployment-dbdump | [releng] | 
            
  | 21:12 | <beta-logmsgbot> | petrb: updating ArticleFeedbackv5 extension | [releng] | 
            
  | 19:51 | <hashar> | 369 languages rebuilt out of 369 | [releng] | 
            
  | 19:45 | <hashar> | rebuilding l10n cache:   mwscript rebuildLocalisationCache.php --wiki=aawiki --threads=2 | [releng] | 
            
  | 19:39 | <hashar> | beta broken by PAGEID magic word introduced with 0a7cf03 / I11d42ca7 {{gerrit|9858}} | [releng] | 
            
  | 19:32 | <hashar> | running git bisect of core 80fbb70..ef3132f | [releng] | 
            
  | 19:26 | <hashar> | upgrading MediaWiki core 80fbb70..ef3132f | [releng] | 
            
  | 19:20 | <hashar> | updated AFTv5: f97811f..d3bd97f | [releng] | 
            
  | 19:04 | <hashar> | updated robots.txt to specify a user-agent. Will definitely prevents Google from killing beta :) | [releng] | 
            
  | 18:54 | <hashar> | squid resumed. The swap files got corrupted somehow, needed to delete them entirely to start again. Squud storing again. | [releng] | 
            
  | 18:40 | <hashar> | -squid bah doing rm -fR /data/project/squid1/* | [releng] | 
            
  | 18:39 | <hashar> | installed `tree` on deployment-squid | [releng] | 
            
  | 18:38 | <hashar> | removing swap files in /data/project/squid1 | [releng] | 
            
  | 18:36 | <hashar> | Squid is bugged as hell : 2012/07/17 18:36:13| Store rebuilding is -0.1% complete and looping | [releng] | 
            
  | 18:21 | <beta-logmsgbot> | hashar: rebooting squid glusterfs gone wild apparently | [releng] | 
            
  | 14:45 | <hashar> | Blacklisted user agents matchin /.*Googlebot.*/ | [releng] | 
            
  | 13:45 | <hashar> | Manually restarted apaches | [releng] | 
            
  | 13:44 | <hashar> | Imported all.conf apache conf from production | [releng] | 
            
  | 13:29 | <hashar> | err: /Stage[main]/Mediawiki::Sync/Exec[mw-sync]: Failed to call refresh: Command exceeded timeout at /etc/puppet/manifests/mediawiki.pp:24 | [releng] | 
            
  | 13:28 | <hashar> | All apaches are dead :/ | [releng] | 
            
  | 09:26 | <hashar> | Adding class role::applicationserver::jobrunner | [releng] | 
            
  | 09:20 | <hashar> | sync upload6 dirs again. root@deployment-nfs-memc:$ rsync -a --progress --inplace /mnt/export/upload6 /data/project/upload6 | [releng] | 
            
  
    | 2012-07-11
      
      § | 
    
  | 16:04 | <hashar> | Created upload.beta.wmflabs.org to points to deployment-cache-upload01 ( 208.80.153.242 ) | [releng] | 
            
  | 15:46 | <hashar> | just logging the rsync command:  root@deployment-nfs-memc:/data/project/upload6# rsync -a --progress --inplace /mnt/export/upload6 /data/project/upload6 | [releng] | 
            
  | 15:41 | <hashar> | started rsync of /mnt/export/upload6 to /data/project/upload6 | [releng] | 
            
  | 14:55 | <hashar> | deployment-cache-bits02 now serves bits.beta.wmflabs.org using pending gerrit changes {{gerrit|15445}} and {{gerrit|13304}} (using varnish configuration from production) | [releng] | 
            
  | 12:51 | <hashar> | created deployment-cache-upload01 a Lucid instane to serve http://upload.beta.wmflabs.org/ | [releng] | 
            
  | 12:49 | <hashar> | deleting cache-upload2 , need a Lucid instance. | [releng] | 
            
  | 12:48 | <hashar> | cache-upload2 set squid_coss_disks to vdb | [releng] | 
            
  | 12:44 | <hashar> | applying role::cache::upload to deployment-cache-upload02 | [releng] | 
            
  | 12:27 | <hashar> | Creating deployment-cache-upload02 to replace deployment-cache-upload and serves http://upload.beta.wmflabs.org/ | [releng] | 
            
  | 12:21 | <hashar> | Deleting deployment-cache-bits which was corrupted and replaced it by deployment-cache-bits02 | [releng] | 
            
  | 12:18 | <hashar> | moved bits.beta.wmflabs.org from deployment-cache-bits to deployment-cache-bits02 | [releng] | 
            
  | 12:09 | <hashar> | Enabling puppetmaster::self on deployment-cache-bits02 to get Varnish config ({{gerrit|13304}}) | [releng] | 
            
  | 09:55 | <hashar> | Applying role::cache::bits::labs to deployment-cache-bits02 | [releng] | 
            
  | 09:42 | <hashar> | Manually started mw-job-runner on jobrunner06 | [releng] | 
            
  | 09:41 | <hashar> | pointed common/php from 'php-1.20wmf6' to 'php-master' | [releng] | 
            
  | 09:25 | <hashar> | updated mediawiki-config on dbdump to latest version | [releng] | 
            
  | 08:55 | <hashar> | add applicationserver::homeless to jobrunner06 | [releng] |