|
2014-03-31
§
|
| 20:00 |
<hashar> |
stopped parsoid . It is killing the application servers |
[releng] |
| 19:53 |
<hashar> |
restarting both apaches |
[releng] |
| 19:21 |
<hashar> |
restarting job service on jobrunner01 to apply {{gerrit|122436}} |
[releng] |
| 19:20 |
<hashar> |
Unbreak puppetmaster on deployment-salt.eqiad.wmflabs |
[releng] |
| 19:01 |
<hashar> |
puppet master is broken :( |
[releng] |
| 17:39 |
<hashar> |
lowering # of jobs spawned by the jobrunner {{gerrit|122436}} |
[releng] |
| 16:00 |
<bd808> |
Restarted logstash service on deployment-logstash1; no new log events seen since 2014-03-28T10:57 |
[releng] |
| 15:58 |
<bd808> |
Updated kibana on deployment-logstash1 to e317bc6 |
[releng] |
| 15:56 |
<hashar_> |
Cluster slow because some CirrusSearch job is spamming simplewiki . Gotta find a way to throttle the number of jobs being run on jobrunner01 or add more apache boxes . It is transient anyway, might look at limiting the runs tonight |
[releng] |
| 15:10 |
<hashar_> |
Rebased puppet repository. Only one hack left: https://gerrit.wikimedia.org/r/#/c/119534/ |
[releng] |
| 14:20 |
<hashar> |
deleting deployment-parsoidcache01 cache the hardway: stopping varnish, deleting files in /srv/vdb/ , starting varnish |
[releng] |
| 14:05 |
<hashar> |
shutdowning database and apache boxes for now. |
[releng] |
| 14:03 |
<hashar> |
shutdowning varnishes instances in pmtpa |
[releng] |
| 13:56 |
<hashar> |
Deleted deployment-cache-upload01 , replaced by deployment-cache-upload02 |
[releng] |
| 13:52 |
<hashar> |
upload varnish cache working :-] |
[releng] |
| 13:47 |
<hashar> |
applying role::cache::upload to role-cache-upload02 |
[releng] |
| 13:37 |
<hashar> |
migrating deployment-cache-upload02.eqiad.Wmflabs to self puppet/salt master |
[releng] |
| 13:22 |
<hashar> |
Creating deployment-cache-upload02 to replace deployment-cache-upload01 which was missing the security group "web" |
[releng] |
| 11:30 |
<hashar> |
Update DNS entries to point to EQIAD instances (aka switching beta cluster to eqiad) |
[releng] |
|
2014-03-27
§
|
| 15:23 |
<hashar> |
role::beta::natfix cant run on deployment-bastion.eqiad because the ferm rules conflicts with the Augeas rules coming from udp2log :-( |
[releng] |
| 15:21 |
<hashar> |
applying role::beta::natfix on deployment-bastion.eqiad |
[releng] |
| 14:58 |
<hashar> |
fixed up role::beta::natfix . Ferm is now being applied again on various application server instances {{gerrit|121378}} |
[releng] |
| 13:58 |
<hashar> |
rebased puppetmaster git repository, reapplied ottomata live hacks. |
[releng] |
| 12:55 |
<hashar> |
mediawiki l10n cache being rebuild!!! |
[releng] |
| 12:54 |
<hashar> |
Fixed permissions on eqiad bastion for /srv/scap . Others (such as mwdeploy) could not read / execute scap scripts |
[releng] |
| 11:29 |
<hashar> |
MediaWiki code and configuration are now self updating on EQIAD cluster via Jenkins jobs. First run: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/4/console |
[releng] |
| 11:11 |
<hashar> |
deleting job beta-code-update , replaced by datacenter variants beta-code-update-pmtpa and beta-code-update-eqiad |
[releng] |
| 10:54 |
<hashar> |
Deleting job beta-update-databases , replaced by datacenter variants beta-update-databases-pmtpa and beta-update-databases-eqiad |
[releng] |
|
2014-03-26
§
|
| 19:05 |
<bd808> |
Added ottomata as a project member and admin |
[releng] |
| 15:46 |
<springle> |
deployment-db1 data loaded |
[releng] |
| 14:45 |
<bd808> |
created proxy https://logstash-beta.wmflabs.org for logstash instance |
[releng] |
| 14:17 |
<hashar> |
fixed up redis configuration in eqiad. Jobrunner is happy now: aawiki-504cd7d2: 0.9649 21.5M Creating a new RedisConnectionPool instance with id 627014dc7020485d721532dde4142d5190ba3cc1. {{gerrit|121060}} |
[releng] |
| 14:05 |
<hashar> |
udp2log functional on eqiad beta cluster \\O/ |
[releng] |
| 13:55 |
<hashar> |
stopping udp2log on eqiad bastion, starting udp2log-mw (really should fix that issue one day) |
[releng] |
| 13:52 |
<hashar> |
dropped some live hack on eqiad in /data/project/apache/common-local and ran git pull |
[releng] |
| 13:14 |
<hashar> |
Dropping enwikivoyage and dewikivoyage databases from sql02. Related changes are updating the Jenkins config: https://gerrit.wikimedia.org/r/#/c/121045/ and cleaning up the mw-config : https://gerrit.wikimedia.org/r/#/c/121047/ |
[releng] |
| 07:53 |
<springle> |
installed mariadb via puppet on deployment-db1. no data yet |
[releng] |