4001-4050 of 5703 results (27ms)
2014-09-03 §
15:02 <bd808> _joe_ rolled out a new hhvm package ~5 hours ago [releng]
15:01 <bd808> morebots is back thanks to petan [releng]
2014-09-02 §
15:34 <bd808> False alarm. SSL is borked in beta and we know that [releng]
15:29 <bd808> `curl -vL -H 'Host: en.wikipedia.beta.wmflabs.org' localhost` works from deployment-cache-text02 [releng]
15:27 <bd808> https://en.wikipedia.beta.wmflabs.org/ returning ERR_CONNECTION_REFUSED (is varnish down?) [releng]
2014-08-29 §
22:56 <bd808> Got puppet to run cleanly on deployment-mediawiki03. Should be ready for serving traffic. [releng]
22:39 <bd808> Fixed a merge conflict in operations/puppet on deployment-salt [releng]
21:46 <bd808> Forced install of "right version of libvips-tools on mediawiki03 `sudo apt-get install libvips-tools=7.38.5-2` [releng]
08:40 <hashar> rebooting deployment-cache-mobile03 (kernel up) [releng]
2014-08-28 §
21:32 <bd808> Added "Greg Grossmeier" to UnderNDA sudoers group [releng]
17:12 <bd808> Changed centralauth db to rename labswiki -> deploymentwiki [releng]
16:49 <bd808> CentralAuth looks broken on http://deployment.wikimedia.beta.wmflabs.org/ [releng]
16:49 <bd808> Apache vhosts look good again [releng]
16:34 <bd808> Restarted varnishes on deployment-cache-text02 [releng]
16:13 <andrewbogott> merging a patch that renames 'labswiki' to 'deploymentwiki' [releng]
09:21 <hashar> resetting git repository in /data/project/apache/conf to point to the betaclusterbranch of operations/mediawiki-config.git discarded all local hacks in the process [releng]
2014-08-27 §
23:03 <hashar> Blacklisting the security audit IP again on deployment-cache bits01 mobile03 and text02 [releng]
22:53 <hashar> removed the blackhole ip route from deployment-cache-text02 and deployment-cache-mobile03 [releng]
22:48 <hashar> the IP is a known security audit. See Chris Steipp. [releng]
22:46 <hashar> blackholed an IP address on deployment-cache-text02 and deployment-cache-mobile03 , it was causing hundred of requests per seconds and overloaded the beta cluster. Use route -n to find the IP [releng]
22:37 <hashar> restarting udp2log-mw on deployment-bastion. It keeps crashing since fiarly recently [releng]
22:26 <bd808> when restarting varnish on deployment-cache-text02, don't forget that there are 2 varnish services (varnish and varnish-frontend) [releng]
22:19 <bd808> restarted varnish (again) on deployment-cache-text02 [releng]
22:10 <bd808> restarted varnish on deployment-cache-text02 [releng]
16:22 <bd808> killing `apt-get update` process running on deployment-bastion since Jun13 [releng]
14:59 <bd808> Resolved puppet git merge conflict on deployment-salt [releng]
14:49 <bd808> Moved hhvm core dumps to /data/project/hhvm-cores [releng]
14:42 <bd808> Root dirve full on deployment-mediawiki02; hhvm core files are the culprit [releng]
2014-08-25 §
23:47 <ori> stopping hhvm/apache on deployment-mediawiki02 to replace debug build of hhvm with release build [releng]
21:44 <bd808> Deployed scap 116027f (Make sync-common update l10n cdb files by default) [releng]
18:30 <ori> deployment-mediawiki02: cleared /tmp; running puppet [releng]
15:05 <hashar> mediawiki02 rm /tmp/hhvm*.core . Filled as {{bug|69979}} [releng]
15:01 <hashar> mediawiki02 rm /tmp/mw-cache-master/conf* [releng]
15:01 <hashar> mediawiki02 has mw conf caches under /tmp/mw-cache-master/ and since that partition is filled up, that ends up with conf caches being null file [releng]
15:00 <hashar> mediawiki02 rm /var/log/upstart/hhvm* [releng]
14:53 <hashar> mediawiki02 : removed /var/lib/puppet/state/agent_catalog_run.lock [releng]
14:46 <hashar> restarting udp2log-mw service on -bastion. It is stalled for some reason [releng]
14:42 <hashar> on mediawiki02 , clearing out some /var/log/upstart/hhvm.* log files see {{bug|69976}} [releng]
14:34 <hashar> mediawiki02 / partition is 100% full [releng]
2014-08-22 §
20:21 <hashar> udp2log are back in /data/project/logs . The udp2log-mw service went stall for some reason. [releng]
20:08 <ori> ran 'git pull' on deployment-salt:/srv/var-lib/git/operations/puppet [releng]
19:59 <hashar> restarting udp2log-mw service on deployment-bastion [releng]
19:59 <hashar> bits yielding 503 [releng]
00:41 <bd808> cherry-picked scap change https://gerrit.wikimedia.org/r/#/c/155677/ for testing [releng]
2014-08-21 §
21:49 <bd808> Trebuchet happier after all the salt-minion restarts; still have deleted hosts showing in the expected minion list for scap deploys [releng]
21:01 <twentyafterfour> Started salt-minion on deployment-redis01 [releng]
21:01 <bd808> Started salt-minon on deployment-upload [releng]
21:00 <bd808> Started salt-minon on deployment-fluoride [releng]
21:00 <bd808> Started salt-minon on deployment-db1 [releng]
20:59 <bd808> Started salt-minon on deployment-elastic01 [releng]