| 2014-03-27
      
      § | 
    
  | 15:23 | <hashar> | role::beta::natfix cant run on deployment-bastion.eqiad because the ferm rules conflicts with the Augeas rules coming from udp2log :-( | [releng] | 
            
  | 15:21 | <hashar> | applying role::beta::natfix on deployment-bastion.eqiad | [releng] | 
            
  | 14:58 | <hashar> | fixed up role::beta::natfix . Ferm is now being applied again on various application server instances {{gerrit|121378}} | [releng] | 
            
  | 13:58 | <hashar> | rebased puppetmaster git repository, reapplied ottomata live hacks. | [releng] | 
            
  | 12:55 | <hashar> | mediawiki l10n cache being rebuild!!! | [releng] | 
            
  | 12:54 | <hashar> | Fixed permissions on eqiad bastion for /srv/scap . Others (such as mwdeploy) could not read / execute scap scripts | [releng] | 
            
  | 11:29 | <hashar> | MediaWiki code and configuration are now self updating on EQIAD cluster via Jenkins jobs. First run: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/4/console | [releng] | 
            
  | 11:11 | <hashar> | deleting job beta-code-update , replaced by datacenter variants beta-code-update-pmtpa and beta-code-update-eqiad | [releng] | 
            
  | 10:54 | <hashar> | Deleting job beta-update-databases , replaced by datacenter variants beta-update-databases-pmtpa and beta-update-databases-eqiad | [releng] | 
            
  
    | 2014-03-26
      
      § | 
    
  | 19:05 | <bd808> | Added ottomata as a project member and admin | [releng] | 
            
  | 15:46 | <springle> | deployment-db1 data loaded | [releng] | 
            
  | 14:45 | <bd808> | created proxy https://logstash-beta.wmflabs.org for logstash instance | [releng] | 
            
  | 14:17 | <hashar> | fixed up redis configuration in eqiad. Jobrunner is happy now: aawiki-504cd7d2: 0.9649  21.5M  Creating a new RedisConnectionPool instance with id 627014dc7020485d721532dde4142d5190ba3cc1. {{gerrit|121060}} | [releng] | 
            
  | 14:05 | <hashar> | udp2log functional on eqiad beta cluster \\O/ | [releng] | 
            
  | 13:55 | <hashar> | stopping udp2log on eqiad bastion, starting udp2log-mw (really should fix that issue one day) | [releng] | 
            
  | 13:52 | <hashar> | dropped some live hack on eqiad in /data/project/apache/common-local and ran git pull | [releng] | 
            
  | 13:14 | <hashar> | Dropping enwikivoyage and dewikivoyage databases from sql02.  Related changes are updating the Jenkins config: https://gerrit.wikimedia.org/r/#/c/121045/ and cleaning up the mw-config : https://gerrit.wikimedia.org/r/#/c/121047/ | [releng] | 
            
  | 07:53 | <springle> | installed mariadb via puppet on deployment-db1. no data yet | [releng] | 
            
  
    | 2014-03-24
      
      § | 
    
  | 23:16 | <marktraceur> | Touching all the MMV scripts because they're not getting invalidated or something | [releng] | 
            
  | 23:10 | <hashar> | l10n cache got broken due to a PHP fatal error I introduced. It is back up now.   Found out via https://integration.wikimedia.org/dashboard/ | [releng] | 
            
  | 23:09 | <hashar> | upgraded all pmtpa varnishes, ran puppet on all of them. all set! | [releng] | 
            
  | 22:57 | <hashar> | restarting deployment-cache-upload04  , apparently stalled\t | [releng] | 
            
  | 22:48 | <hashar> | upgrading varnish on all pmtpa caches. | [releng] | 
            
  | 22:47 | <hashar> | apt-get upgrade varnish on deployment-cache-bits03 | [releng] | 
            
  | 22:45 | <marktraceur> | attempted restart of varnish on betalabs; seems to have failed, trying again | [releng] | 
            
  | 22:42 | <hashar> | made marktraceur a project admin and granted sudo rights | [releng] | 
            
  | 22:39 | <marktraceur> | Restarting betalabs varnish to workaround https://bugzilla.wikimedia.org/show_bug.cgi?id=63034 | [releng] | 
            
  | 17:25 | <bd808> | Converted deployment-db1.eqiad.wmflabs to use local puppet & salt masters | [releng] | 
            
  | 17:06 | <bd808> | Changed rules in sql security group to use CIDR 10.0.0.0/8. | [releng] | 
            
  | 17:05 | <bd808> | Changed rules in search security group to use CIDR 10.0.0.0/8. | [releng] | 
            
  | 17:05 | <bd808> | Built deployment-elastic04.eqiad.wmflabs with local salt/puppet master, secondary disk on /var/lib/elasticsearch and role::elasticsearch::server | [releng] | 
            
  | 16:19 | <bd808> | Built deployment-elastic03.eqiad.wmflabs with local salt/puppet master, secondary disk on /var/lib/elasticsearch and role::elasticsearch::server | [releng] | 
            
  | 16:08 | <bd808> | Built deployment-elastic02.eqiad.wmflabs with local salt/puppet master, secondary disk on /var/lib/elasticsearch and role::elasticsearch::server | [releng] | 
            
  | 15:54 | <bd808> | Built deployment-elastic01.eqiad.wmflabs with local salt/puppet master, secondary disk on /var/lib/elasticsearch and role::elasticsearch::server | [releng] | 
            
  | 10:31 | <hashar> | migrated deployment-solr to self puppet/salt masters | [releng] |