| 2015-06-25
      
      § | 
    
  | 11:03 | <hashar> | Rebooting  integration-raita and integration-vmbuilder-trusty | [releng] | 
            
  | 11:01 | <hashar> | Unmounting /data/project and /home NFS mounts from integration-raita and integration-vmbuilder-trusty https://phabricator.wikimedia.org/T90610 | [releng] | 
            
  | 10:45 | <hashar> | deployment-sca02 deleted /var/lib/puppet/state/agent_catalog_run.lock from June 5th | [releng] | 
            
  | 08:57 | <hashar> | Fixed puppet "Can't dup Symbol" on deployment-pdf01  by deleting puppet, /var/lib/puppet and reinstalling it from scratch https://phabricator.wikimedia.org/T87197 | [releng] | 
            
  | 08:39 | <hashar> | apt-get upgrade deployment-salt | [releng] | 
            
  | 08:08 | <hashar> | deployment-pdf01 deleted /var/log/ocg/ content. Last entry is from July 25th 2014 and puppet complains with <tt>e[/var/log/ocg]: Not removing directory; use 'force' to override</tt> | [releng] | 
            
  | 08:05 | <hashar> | apt-get upgrade deployment-pdf01 | [releng] | 
            
  | 06:37 | <Krinkle> | Reloading Zuul to deploy https://gerrit.wikimedia.org/r/220712 | [releng] | 
            
  | 06:33 | <Krinkle> | Reloading Zuul to deploy https://gerrit.wikimedia.org/r/220705 | [releng] | 
            
  
    | 2015-06-24
      
      § | 
    
  | 19:31 | <hashar> | rebooting deployment-cache-upload02 | [releng] | 
            
  | 19:28 | <hashar> | fixing DNS puppet etc on deployment-cache-upload02 | [releng] | 
            
  | 19:24 | <hashar> | rebooting deployment-zookeeper to get rid of the /home NFS https://phabricator.wikimedia.org/T102169 | [releng] | 
            
  | 19:06 | <hashar> | beta: salt 'i-00*' cmd.run "echo 'domain integration.eqiad.wmflabs\\nsearch integration.eqiad.wmflabs eqiad.wmflabs\\nnameserver 208.80.154.20\\noptions timeout:5' > /etc/resolv.conf" | [releng] | 
            
  | 19:06 | <hashar> | fixing DNS / puppet and salt on i-000008d5.eqiad.wmflabs  i-000002de.eqiad.wmflabs i-00000958.eqiad.wmflabs | [releng] | 
            
  | 15:35 | <hashar> | integration-dev recovered!  puppet hasn't run for ages but caught up with changes | [releng] | 
            
  | 15:13 | <hashar> | removed /var/lib/puppet/state/agent_catalog_run.lock on integration-dev | [releng] | 
            
  | 09:52 | <hashar> | Java 6 removed from gallium / lanthanum and CI labs slaves. | [releng] | 
            
  | 09:18 | <hashar> | getting rid of java 6 on CI machines ( https://phabricator.wikimedia.org/T103491 ) | [releng] | 
            
  | 07:58 | <hashar> | Bah puppet reenable NFS on deployment-parsoidcache02 for some reason | [releng] | 
            
  | 07:57 | <hashar> | disabling NFS on deployment-parsoidcache02 | [releng] | 
            
  | 00:38 | <marxarelli> | reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/219513/ | [releng] | 
            
  | 00:32 | <marxarelli> | running `jenkins-jobs update` to create 'mwext-MobileFrontend-mw-selenium' with I7affe14e878d5c1fc4bcb4dfc7f2d1494cd795b7 | [releng] | 
            
  | 00:20 | <marxarelli> | running `jenkins-jobs update` to create 'mediawiki-selenium-integration' with I7affe14e878d5c1fc4bcb4dfc7f2d1494cd795b7 | [releng] | 
            
  
    | 2015-06-23
      
      § | 
    
  | 23:29 | <Krinkle> | Reloading Zuul to deploy https://gerrit.wikimedia.org/r/220350 | [releng] | 
            
  | 21:34 | <bd808> | updated scap to 33f3002 (Ensure that the minimum batch size used by cluster_ssh is 1) | [releng] | 
            
  | 19:53 | <legoktm> | deleted broken renames from centralauth.renameuser_status on beta cluster | [releng] | 
            
  | 18:28 | <jzerebecki> | zuul reload for https://gerrit.wikimedia.org/r/#/c/219778/4 | [releng] | 
            
  | 16:33 | <bd808> | updated scap to da64a65 (Cast pid read from file to an int) | [releng] | 
            
  | 16:20 | <bd808> | updated scap to 947b93f (Fix reference to _get_apache_list) | [releng] | 
            
  | 13:31 | <cscott> | fixed salt on deployment-pdf02, restarted OCG there. | [releng] | 
            
  | 12:24 | <hashar> | rebooting integration-labvagrant (stuck) | [releng] | 
            
  | 05:44 | <cscott> | stopped OCG service on deployment-pdf02, see https://phabricator.wikimedia.org/T103473 | [releng] | 
            
  | 05:20 | <cscott> | updated OCG to version d7c698d5bf730d34057945e912ac75dc542dd788 ; restarted service. | [releng] | 
            
  | 03:58 | <cscott> | stopped OCG on beta; redis 2.8.x is causing the service to crash on startup. | [releng] | 
            
  | 00:07 | <legoktm> | deploying https://gerrit.wikimedia.org/r/220020 | [releng] | 
            
  
    | 2015-06-22
      
      § | 
    
  | 22:23 | <legoktm> | deploying https://gerrit.wikimedia.org/r/219603 | [releng] | 
            
  | 21:58 | <andrewbogott> | re-enabling puppet on deployment-videoscaler01 because no reason was given for disabling | [releng] | 
            
  | 21:47 | <bd808> | scap emitting soft failures due to missing python-netifaces on deployment-videoscaler01; should be fixed by a current puppet run | [releng] | 
            
  | 21:37 | <bd808> | Updated scap to 81b7c14 (Move dsh group file names to config) | [releng] | 
            
  | 20:42 | <cscott> | updated OCG to version b482144f5bd8b427bcc64a3dd287247195aa1951 | [releng] | 
            
  | 14:58 | <hashar> | disabled  sshd MAC/KEX hardening on beta (was https://gerrit.wikimedia.org/r/#/c/219828/ ) | [releng] | 
            
  | 14:32 | <hashar> | restarting Jenkins | [releng] | 
            
  | 14:30 | <hashar> | Reenable sshd MAC/KEX hardening on beta by cherry picking https://gerrit.wikimedia.org/r/#/c/219828/ | [releng] | 
            
  | 13:17 | <moritzm> | activated firejail service containment for graphoid, citoid and mathoid in deployment-sca | [releng] | 
            
  | 11:07 | <hashar> | fixing puppet on integration-zuul-server | [releng] | 
            
  | 10:29 | <hashar> | rebooted deployment-kafka02 to get rid of /home NFS share | [releng] | 
            
  | 10:25 | <hashar> | fixed puppet.conf on deployment-urldownloader | [releng] | 
            
  | 10:20 | <hashar> | enabled puppet agent on deployment-urldownloader | [releng] | 
            
  | 10:05 | <hashar> | removing puppet lock on deployment-elastic07 ( rm /var/lib/puppet/state/agent_catalog_run.lock ) | [releng] | 
            
  | 09:40 | <hashar> | fixed puppet certificates on integration-lightslave-jessie-1002 by deleting the SSL certs | [releng] |