|
2014-09-10
§
|
| 19:19 |
<bd808> |
Deleted /var/log/account/pacct* and /var/log/atop.log.* on deployment-jobrunner01 to make some temporary room in /var |
[releng] |
| 19:14 |
<bd808> |
Deleted /var/log/mediawiki/jobrunner.log and restarted jobrunner on deployment-jobrunner01: |
[releng] |
| 19:11 |
<bd808> |
/var full on deployment-jobrunner01 |
[releng] |
| 19:05 |
<bd808> |
Deleted /srv/common-local on deployment-jobrunner01 |
[releng] |
| 19:04 |
<bd808> |
Changed /usr/local/apache/common-local symlink to point to /srv/mediawiki on deployment-jobrunner01 |
[releng] |
| 19:03 |
<bd808> |
w00t!!! scap jobs is green again -- https://integration.wikimedia.org/ci/job/beta-scap-eqiad/20965/ |
[releng] |
| 19:00 |
<bd808> |
sync-common finished on deployement-jobrunner01; trying Jenkins scap job again |
[releng] |
| 18:53 |
<bd808> |
Removed symlink and make /srv/mediawiki a proper directory on deployment-jobrunner01; Running sync-common to populate. |
[releng] |
| 18:45 |
<bd808> |
Made /srv/mediawiki a symling to /srv/common-local on deployment-jobrunner01 |
[releng] |
| 10:20 |
<jeremyb> |
deployment-bastion /var at 97%, freed up ~500MB. apt-get clean && rm -rv /var/log/account/pacct* |
[releng] |
| 10:17 |
<jeremyb> |
deployment-bastion good puppet run |
[releng] |
| 10:16 |
<jeremyb> |
deployment-salt had an oom-kill recently. and some box (maybe master, maybe client?) had a disk fill up |
[releng] |
| 10:15 |
<jeremyb> |
deployment-mediawiki0[12] both had good puppet runs |
[releng] |
| 10:15 |
<jeremyb> |
deployment-salt started puppetmaster && puppet run |
[releng] |
| 10:14 |
<jeremyb> |
deployment-bastion killed puppet lock |
[releng] |
| 08:14 |
<Krinkle> |
bits.beta.wmflabs.org is down with 503 Service Unavailable (http://bits.beta.wmflabs.org/en.wikipedia.beta.wmflabs.org/load.php) |
[releng] |
| 03:04 |
<bd808> |
Ori made puppet changes that moved the MediaWiki install dir to /srv/mediawiki (https://gerrit.wikimedia.org/r/#/c/159431/). I didn't see that in SAL so I'm adding it here. |
[releng] |
|
2014-09-04
§
|
| 16:06 |
<bd808> |
Manually cleaned bogus LocalRenameUserJob jobs from redis |
[releng] |
| 13:54 |
<_joe_> |
stopped puppet on the appservers but mw03, testing an apache change |
[releng] |
| 05:28 |
<legoktm> |
stopping jobrunner on deployment-jobrunner01 |
[releng] |
| 05:22 |
<legoktm> |
restarted jobrunner on deployment-jobrunner01 |
[releng] |
| 05:14 |
<bd808> |
Bad jobs in job queue filled up /var on jobrunner01 and killed jobrunner script. Leaving down for now until I find out how to delete the bad jobs. |
[releng] |
| 01:41 |
<bd808> |
Killed old jobs-loop.sh processes on deployment-jobrunner01 |
[releng] |
| 01:24 |
<bd808> |
Many jobrunner errors like "wikiversions-labs.cdb has no version entry for `amwiki`" with various wiki names |
[releng] |
| 01:23 |
<bd808|AWAY> |
Started jobrunner service manually on jobrunner01. |
[releng] |
| 00:44 |
<bd808> |
Puppet run on deployment-jobrunner01 failing with what seem to be dns issues (getaddrinfo: Name or service not known when Trebuchet is running) |
[releng] |
| 00:35 |
<bd808> |
Puppet run on deployment-jobrunner01 failing with what seem to be dns issues (getaddrinfo: Name or service not known) |
[releng] |