| 
      
        2014-03-31
      
      §
     | 
  
    
  | 15:10 | 
  <hashar_> | 
  Rebased puppet repository. Only one hack left: https://gerrit.wikimedia.org/r/#/c/119534/ | 
  [releng] | 
            
  | 14:20 | 
  <hashar> | 
  deleting deployment-parsoidcache01 cache the hardway: stopping varnish, deleting files in /srv/vdb/ , starting varnish | 
  [releng] | 
            
  | 14:05 | 
  <hashar> | 
  shutdowning database and apache boxes for now. | 
  [releng] | 
            
  | 14:03 | 
  <hashar> | 
  shutdowning varnishes instances in pmtpa | 
  [releng] | 
            
  | 13:56 | 
  <hashar> | 
  Deleted deployment-cache-upload01 , replaced by deployment-cache-upload02 | 
  [releng] | 
            
  | 13:52 | 
  <hashar> | 
  upload varnish cache working :-] | 
  [releng] | 
            
  | 13:47 | 
  <hashar> | 
  applying role::cache::upload to role-cache-upload02 | 
  [releng] | 
            
  | 13:37 | 
  <hashar> | 
  migrating deployment-cache-upload02.eqiad.Wmflabs to self puppet/salt master | 
  [releng] | 
            
  | 13:22 | 
  <hashar> | 
  Creating deployment-cache-upload02 to replace deployment-cache-upload01 which was missing the security group "web" | 
  [releng] | 
            
  | 11:30 | 
  <hashar> | 
  Update DNS entries to point to EQIAD instances (aka switching beta cluster to eqiad) | 
  [releng] | 
            
  
    | 
      
        2014-03-27
      
      §
     | 
  
    
  | 15:23 | 
  <hashar> | 
  role::beta::natfix cant run on deployment-bastion.eqiad because the ferm rules conflicts with the Augeas rules coming from udp2log :-( | 
  [releng] | 
            
  | 15:21 | 
  <hashar> | 
  applying role::beta::natfix on deployment-bastion.eqiad | 
  [releng] | 
            
  | 14:58 | 
  <hashar> | 
  fixed up role::beta::natfix . Ferm is now being applied again on various application server instances {{gerrit|121378}} | 
  [releng] | 
            
  | 13:58 | 
  <hashar> | 
  rebased puppetmaster git repository, reapplied ottomata live hacks. | 
  [releng] | 
            
  | 12:55 | 
  <hashar> | 
  mediawiki l10n cache being rebuild!!! | 
  [releng] | 
            
  | 12:54 | 
  <hashar> | 
  Fixed permissions on eqiad bastion for /srv/scap . Others (such as mwdeploy) could not read / execute scap scripts | 
  [releng] | 
            
  | 11:29 | 
  <hashar> | 
  MediaWiki code and configuration are now self updating on EQIAD cluster via Jenkins jobs. First run: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/4/console | 
  [releng] | 
            
  | 11:11 | 
  <hashar> | 
  deleting job beta-code-update , replaced by datacenter variants beta-code-update-pmtpa and beta-code-update-eqiad | 
  [releng] | 
            
  | 10:54 | 
  <hashar> | 
  Deleting job beta-update-databases , replaced by datacenter variants beta-update-databases-pmtpa and beta-update-databases-eqiad | 
  [releng] | 
            
  
    | 
      
        2014-03-26
      
      §
     | 
  
    
  | 19:05 | 
  <bd808> | 
  Added ottomata as a project member and admin | 
  [releng] | 
            
  | 15:46 | 
  <springle> | 
  deployment-db1 data loaded | 
  [releng] | 
            
  | 14:45 | 
  <bd808> | 
  created proxy https://logstash-beta.wmflabs.org for logstash instance | 
  [releng] | 
            
  | 14:17 | 
  <hashar> | 
  fixed up redis configuration in eqiad. Jobrunner is happy now: aawiki-504cd7d2: 0.9649  21.5M  Creating a new RedisConnectionPool instance with id 627014dc7020485d721532dde4142d5190ba3cc1. {{gerrit|121060}} | 
  [releng] | 
            
  | 14:05 | 
  <hashar> | 
  udp2log functional on eqiad beta cluster \\O/ | 
  [releng] | 
            
  | 13:55 | 
  <hashar> | 
  stopping udp2log on eqiad bastion, starting udp2log-mw (really should fix that issue one day) | 
  [releng] | 
            
  | 13:52 | 
  <hashar> | 
  dropped some live hack on eqiad in /data/project/apache/common-local and ran git pull | 
  [releng] | 
            
  | 13:14 | 
  <hashar> | 
  Dropping enwikivoyage and dewikivoyage databases from sql02.  Related changes are updating the Jenkins config: https://gerrit.wikimedia.org/r/#/c/121045/ and cleaning up the mw-config : https://gerrit.wikimedia.org/r/#/c/121047/ | 
  [releng] | 
            
  | 07:53 | 
  <springle> | 
  installed mariadb via puppet on deployment-db1. no data yet | 
  [releng] | 
            
  
    | 
      
        2014-03-24
      
      §
     | 
  
    
  | 23:16 | 
  <marktraceur> | 
  Touching all the MMV scripts because they're not getting invalidated or something | 
  [releng] | 
            
  | 23:10 | 
  <hashar> | 
  l10n cache got broken due to a PHP fatal error I introduced. It is back up now.   Found out via https://integration.wikimedia.org/dashboard/ | 
  [releng] | 
            
  | 23:09 | 
  <hashar> | 
  upgraded all pmtpa varnishes, ran puppet on all of them. all set! | 
  [releng] | 
            
  | 22:57 | 
  <hashar> | 
  restarting deployment-cache-upload04  , apparently stalled\t | 
  [releng] | 
            
  | 22:48 | 
  <hashar> | 
  upgrading varnish on all pmtpa caches. | 
  [releng] | 
            
  | 22:47 | 
  <hashar> | 
  apt-get upgrade varnish on deployment-cache-bits03 | 
  [releng] | 
            
  | 22:45 | 
  <marktraceur> | 
  attempted restart of varnish on betalabs; seems to have failed, trying again | 
  [releng] | 
            
  | 22:42 | 
  <hashar> | 
  made marktraceur a project admin and granted sudo rights  | 
  [releng] | 
            
  | 22:39 | 
  <marktraceur> | 
  Restarting betalabs varnish to workaround https://bugzilla.wikimedia.org/show_bug.cgi?id=63034 | 
  [releng] | 
            
  | 17:25 | 
  <bd808> | 
  Converted deployment-db1.eqiad.wmflabs to use local puppet & salt masters | 
  [releng] | 
            
  | 17:06 | 
  <bd808> | 
  Changed rules in sql security group to use CIDR 10.0.0.0/8. | 
  [releng] | 
            
  | 17:05 | 
  <bd808> | 
  Changed rules in search security group to use CIDR 10.0.0.0/8. | 
  [releng] | 
            
  | 17:05 | 
  <bd808> | 
  Built deployment-elastic04.eqiad.wmflabs with local salt/puppet master, secondary disk on /var/lib/elasticsearch and role::elasticsearch::server | 
  [releng] |