3401-3450 of 5481 results (15ms)
2014-12-16 §
16:48 <manybubbles> restarting beta's elasticsearch servers to pick up a new version of a plugin. won't interfere with current downtime. [releng]
13:51 <hashar> deleting integration-slave1001 and recreating it. It is blocked on boot and we can't console on it https://phabricator.wikimedia.org/T76250 [releng]
2014-12-15 §
23:24 <Krinkle> integration-slave1001 isn't coming back (T76250), building integration-slave1005 as its replacement. [releng]
12:53 <YuviPanda> manually restarted diamond on all betalabs host, to see if that is why metrics aren’t being sent anymore [releng]
09:41 <hashar> deleted hhvm core files in /var/tmp/core from both mediawiki01 and mediawiki02 {{T1259}} and {{T71979}} [releng]
2014-12-13 §
18:51 <bd808> Running chmod -R g+s /data/project/upload7 on deploymnet-mediawiki02 [releng]
18:25 <bd808> Running chmod -R u=rwX,g=rwX,o=rX /data/project/upload7 from deployment-mediawiki02 [releng]
18:16 <bd808> chown done for /data/project/upload7 [releng]
17:51 <bd808> Running chown -R apache:apache on /data/project/upload7 from deployment-mediawiki02 [releng]
17:11 <bd808> Labs DNS seems to be flaking out badly and causing random scap and puppet failures [releng]
17:10 <bd808> Many strange puppet and scap failures in beta that look to be related to DNS failures [releng]
16:58 <bd808> restarted puppetmaster on deployment-salt [releng]
16:31 <bd808> apache user renumbered on deployment-mediawiki03 [releng]
16:23 <bd808> apache and hhvm restarted on beta app servers following apache user renumber [releng]
16:09 <bd808> apache and hhvm stopped on beta app server tier. All requests expected to return 503 from varnish [releng]
16:03 <bd808> Starting work on [[phab:T78076]] to renumber apache users in beta [releng]
16:03 <bd808> Starting work on [[phab:T78076]] to renumber apache users in beta [releng]
08:21 <YuviPanda|zzz> forcing puppet run on all deployment-prep hosts [releng]
2014-12-12 §
22:38 <bd808> Fixed scap by deleting /srv/mediawiki/~tmp~ on deployment-rsync01 [releng]
22:27 <hashar> Creating 1300 Jenkins jobs to run extensions PHPUnit tests under either HHVM or Zend PHP flavors. [releng]
18:35 <bd808> Added puppet config to record !log messages in logstash [releng]
17:32 <bd808> forcing puppet runs on deployment-mediawiki0[12]; hiera settings specific to beta were not applied on the hosts leading to all kinds of problems [releng]
17:13 <bd808> restarted hhvm on deployment-mediawiki0[12] and purged hhbc database [releng]
17:00 <bd808> restarted apache2 on deployment-mediawiki01 [releng]
16:59 <bd808> restarted apache2 on deployment-mediawiki02 [releng]
2014-12-11 §
22:47 <cscott> updated OCG to version bfc3812ef346c9f767135b339cedd123a1bcac98 [releng]
22:13 <hashar> Adding chrismcmahon to the 'integration' Gerrit group so he can +2 changes made to integration/config.git [releng]
21:47 <hashar> Jenkins re adding [https://integration.wikimedia.org/ci/computer/integration-slave1009/ integration-slave1009] to the pool of slaves [releng]
19:45 <bd808|LUNCH> I got nerd snipped into looking at beta. Major personal productivity failure. [releng]
19:43 <bd808|LUNCH> nslcd log noise is probably a red herring -- https://access.redhat.com/solutions/58684 [releng]
19:39 <bd808|LUNCH> lots of nslcd errors in syslog on deployment-rsync01 which may be causing scap failures [releng]
07:45 <YuviPanda> shut up shinken-wm [releng]
2014-12-10 §
22:17 <bd808> restarted logstash on logstash1001. redis event queue not being processed [releng]
10:30 <hashar> Adding hhvm on Trusty slaves, using depooled integration-slave1009 as the main work area [releng]
2014-12-09 §
16:33 <bd808> restarted puppetmaster to pick up changes to custom functions [releng]
16:19 <bd808> forced install of sudo-ldap across beta with: salt '*' cmd.run 'env SUDO_FORCE_REMOVE=yes DEBIAN_FRONTEND=noninteractive apt-get -y install sudo-ldap' [releng]
2014-12-08 §
23:45 <bd808> deleted hhvm core on mediawiki01 [releng]
23:43 <bd808> Ran `apt-get clean` on deployment-mediawiki01 [releng]
2014-12-06 §
05:05 <ori> upgrade hhvm-tidy to 0.1-2 [releng]
2014-12-05 §
22:21 <bd808> 1.1G free on deployment-mediawiki02:/var after removing a lot of crap form logs and /var/tmp/cores [releng]
22:06 <bd808> /var full on deployment-mediawiki02 again :((( [releng]
10:50 <hashar> applying mediawiki::multimedia class on contint slaves ( https://phabricator.wikimedia.org/T76661 | https://gerrit.wikimedia.org/r/#/c/177770/ ) [releng]
01:01 <bd808> Deleted a ton of jeprof.*.heap files from deployment-mediawiki02:/ [releng]
00:54 <YuviPanda> cleared out pngs from mediawiki02 to kill low space warning [releng]
00:53 <YuviPanda> mediawiki02 instance is low on space, /tmp has lots of... pngs? [releng]
2014-12-04 §
22:48 <YuviPanda> manually rebased puppet on deployment-prep [releng]
00:29 <bd808> deleted instance "udplog" [releng]
2014-12-03 §
21:33 <cscott> updated OCG to version 08e94b19c3f17e699d7e53d9605f65c58e17ea0e [releng]
19:11 <bd808> Cleaned up legacy jobrunner scripts on deployment-jobrunner01 (/etc/default/mw-job-runner /etc/init.d/mw-job-runner /usr/local/bin/jobs-loop.sh) [releng]
2014-12-02 §
23:39 <bd808> Cause of full disk on deployment-mediawiki01 was an hhvm core file; fixed now [releng]