3651-3700 of 5371 results (21ms)
2014-09-07 §
23:48 <bd808> Added John F. Lewis to under_NDA sudo policy (bug 70539) [releng]
23:29 <bd808> Promoted John F. Lewis to project admin (bug 70539) [releng]
23:26 <bd808> Added Jalexander as project member (bug 70539) [releng]
07:00 <jeremyb> testing 1,2,3 [releng]
2014-09-05 §
17:54 <bd808> Purged varnish cache on deployment-cache-bits01 -- sudo varnishadm ban req.url '~' / [releng]
16:00 <YuviPanda> unfuck puppet on deployment-salt, puppet is stupid and does not properly report failed events on last_run_summary.yaml if there's a syntax error or a resource conflict. So I've to read last_run_report and do things with *that* instead now [releng]
15:49 <YuviPanda> deliberately fucking up puppet to see if icinga complains [releng]
09:52 <_joe_> cherry-picked I6ec53da483bebfa375eba2383cbf60123ff1ce26, it work [releng]
2014-09-04 §
16:06 <bd808> Manually cleaned bogus LocalRenameUserJob jobs from redis [releng]
13:54 <_joe_> stopped puppet on the appservers but mw03, testing an apache change [releng]
05:28 <legoktm> stopping jobrunner on deployment-jobrunner01 [releng]
05:22 <legoktm> restarted jobrunner on deployment-jobrunner01 [releng]
05:14 <bd808> Bad jobs in job queue filled up /var on jobrunner01 and killed jobrunner script. Leaving down for now until I find out how to delete the bad jobs. [releng]
01:41 <bd808> Killed old jobs-loop.sh processes on deployment-jobrunner01 [releng]
01:24 <bd808> Many jobrunner errors like "wikiversions-labs.cdb has no version entry for `amwiki`" with various wiki names [releng]
01:23 <bd808|AWAY> Started jobrunner service manually on jobrunner01. [releng]
00:44 <bd808> Puppet run on deployment-jobrunner01 failing with what seem to be dns issues (getaddrinfo: Name or service not known when Trebuchet is running) [releng]
00:35 <bd808> Puppet run on deployment-jobrunner01 failing with what seem to be dns issues (getaddrinfo: Name or service not known) [releng]
2014-09-03 §
15:02 <bd808> _joe_ rolled out a new hhvm package ~5 hours ago [releng]
15:01 <bd808> morebots is back thanks to petan [releng]
2014-09-02 §
15:34 <bd808> False alarm. SSL is borked in beta and we know that [releng]
15:29 <bd808> `curl -vL -H 'Host: en.wikipedia.beta.wmflabs.org' localhost` works from deployment-cache-text02 [releng]
15:27 <bd808> https://en.wikipedia.beta.wmflabs.org/ returning ERR_CONNECTION_REFUSED (is varnish down?) [releng]
2014-08-29 §
22:56 <bd808> Got puppet to run cleanly on deployment-mediawiki03. Should be ready for serving traffic. [releng]
22:39 <bd808> Fixed a merge conflict in operations/puppet on deployment-salt [releng]
21:46 <bd808> Forced install of "right version of libvips-tools on mediawiki03 `sudo apt-get install libvips-tools=7.38.5-2` [releng]
08:40 <hashar> rebooting deployment-cache-mobile03 (kernel up) [releng]
2014-08-28 §
21:32 <bd808> Added "Greg Grossmeier" to UnderNDA sudoers group [releng]
17:12 <bd808> Changed centralauth db to rename labswiki -> deploymentwiki [releng]
16:49 <bd808> CentralAuth looks broken on http://deployment.wikimedia.beta.wmflabs.org/ [releng]
16:49 <bd808> Apache vhosts look good again [releng]
16:34 <bd808> Restarted varnishes on deployment-cache-text02 [releng]
16:13 <andrewbogott> merging a patch that renames 'labswiki' to 'deploymentwiki' [releng]
09:21 <hashar> resetting git repository in /data/project/apache/conf to point to the betaclusterbranch of operations/mediawiki-config.git discarded all local hacks in the process [releng]
2014-08-27 §
23:03 <hashar> Blacklisting the security audit IP again on deployment-cache bits01 mobile03 and text02 [releng]
22:53 <hashar> removed the blackhole ip route from deployment-cache-text02 and deployment-cache-mobile03 [releng]
22:48 <hashar> the IP is a known security audit. See Chris Steipp. [releng]
22:46 <hashar> blackholed an IP address on deployment-cache-text02 and deployment-cache-mobile03 , it was causing hundred of requests per seconds and overloaded the beta cluster. Use route -n to find the IP [releng]
22:37 <hashar> restarting udp2log-mw on deployment-bastion. It keeps crashing since fiarly recently [releng]
22:26 <bd808> when restarting varnish on deployment-cache-text02, don't forget that there are 2 varnish services (varnish and varnish-frontend) [releng]
22:19 <bd808> restarted varnish (again) on deployment-cache-text02 [releng]
22:10 <bd808> restarted varnish on deployment-cache-text02 [releng]
16:22 <bd808> killing `apt-get update` process running on deployment-bastion since Jun13 [releng]
14:59 <bd808> Resolved puppet git merge conflict on deployment-salt [releng]
14:49 <bd808> Moved hhvm core dumps to /data/project/hhvm-cores [releng]
14:42 <bd808> Root dirve full on deployment-mediawiki02; hhvm core files are the culprit [releng]
2014-08-25 §
23:47 <ori> stopping hhvm/apache on deployment-mediawiki02 to replace debug build of hhvm with release build [releng]
21:44 <bd808> Deployed scap 116027f (Make sync-common update l10n cdb files by default) [releng]
18:30 <ori> deployment-mediawiki02: cleared /tmp; running puppet [releng]
15:05 <hashar> mediawiki02 rm /tmp/hhvm*.core . Filled as {{bug|69979}} [releng]