|
2015-03-30
§
|
| 22:58 |
<legoktm> |
1001-1003 were depooled, restarted and repooled. 1004 is depooled and restarted |
[releng] |
| 22:40 |
<legoktm> |
rebooting precise jenkins slaves |
[releng] |
| 22:33 |
<Josve05a> |
manually start mysql on db1 and db2 |
[releng] |
| 21:57 |
<YuviPanda> |
reboot all instances from virt1000 |
[releng] |
| 21:40 |
<greg-g> |
Beta Cluster is down due to WMF Labs issues, being taken care of now (by Coren and Yuvi) |
[releng] |
| 19:53 |
<legoktm> |
deleted core dumps from integration-slave1001 |
[releng] |
| 19:11 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/200646 |
[releng] |
| 16:29 |
<jzerebecki> |
another damaged git repo integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-qunit/src/vendor/ |
[releng] |
| 16:07 |
<jzerebecki> |
removing workspaces of deleted jobs integration-slave100{1,2,3,4}:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-{client,repo,repo-api}-tests{,@*} |
[releng] |
| 15:14 |
<jzerebecki> |
integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-repo-api-tests-sqlite |
[releng] |
| 15:05 |
<jzerebecki> |
integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-repo-api-tests-mysql/src/extensions/cldr |
[releng] |
| 14:36 |
<jzerebecki> |
integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-*-tests{,@*} |
[releng] |
| 13:06 |
<jzerebecki> |
integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-client-tests@* |
[releng] |
| 13:05 |
<jzerebecki> |
integration-slave1001:~$ sudo -u jenkins-deploy rm -rf /mnt/jenkins-workspace/workspace/mwext-Wikibase-client-tests |
[releng] |
|
2015-03-27
§
|
| 23:28 |
<bd808> |
applied beta::autoupdater directly to deployment-bastion via wikitech interface |
[releng] |
| 23:21 |
<bd808> |
Duplicate declaration: Git::Clone[operations/mediawiki-config] is already declared in file /etc/puppet/modules/beta/manifests/autoupdater.pp:46; cannot redeclare at /etc/puppet/modules/scap/manifests/master.pp:22 |
[releng] |
| 23:01 |
<bd808> |
restarted puppetmaster |
[releng] |
| 22:52 |
<hashar> |
integration: jzerebecki addition and sudo policy tracked for history purpose as {{bug|T94280}} |
[releng] |
| 22:52 |
<bd808> |
chown -R l10nupdate:wikidev /srv/mediawiki-staging/php-master/cache/l10n |
[releng] |
| 22:44 |
<bd808> |
deployment-bastion: chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/ |
[releng] |
| 22:41 |
<bd808> |
forcing puppet run on deployment-bastion |
[releng] |
| 22:41 |
<bd808> |
cherry-picked https://gerrit.wikimedia.org/r/#/c/200248/ and https://gerrit.wikimedia.org/r/#/c/199988/ |
[releng] |
| 22:40 |
<hashar> |
integration: created sudo policy allowing members to run any command as jenkins-deploy on all hosts. |
[releng] |
| 22:40 |
<hashar> |
added jzerebecki to the integration labs project as a normal member |
[releng] |
| 22:34 |
<hashar> |
integration-slave1001 rm -fR mwext-Wikibase-repo-api-tests/src/vendor |
[releng] |
| 21:13 |
<greg-g> |
things be better |
[releng] |
| 20:56 |
<greg-g> |
Beta Cluster is down, known |
[releng] |
| 18:50 |
<marxarelli> |
running `jenkins-jobs update` to update 'browsertests-UploadWizard-*' with Id33ffde07f0c15e153d52388cf130be4c59b4559 |
[releng] |
| 17:50 |
<legoktm> |
deleted core dumps from integration-slave1002 |
[releng] |
| 17:48 |
<legoktm> |
marked integration-slave1002 as offline, /var filled up |
[releng] |
| 05:42 |
<legoktm> |
marked integration-slave1001 as offline due to https://phabricator.wikimedia.org/T94138 |
[releng] |
|
2015-03-26
§
|
| 23:47 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/200069 |
[releng] |
| 19:22 |
<bd808> |
Manually added missing !log entries from 2015-03-25 from my bouncer logs |
[releng] |
| 17:14 |
<greg-g> |
jobs appear to be processing according to zuul, the Jenkins UI just takes forever to load, apparently |
[releng] |
| 17:12 |
<greg-g> |
"Please wait while Jenkins is getting ready to work" |
[releng] |
| 17:08 |
<greg-g> |
0:07 < robh> kill -9 and restarted per instrucitons |
[releng] |
| 16:53 |
<greg-g> |
Still.... "Please wait while Jenkins is restarting..." |
[releng] |
| 16:49 |
<greg-g> |
"Please wait while Jenkins is restarting..." |
[releng] |
| 16:39 |
<greg-g> |
going to do a safe-restart of Jenkins https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Restart_all_of_Jenkins |
[releng] |
| 16:38 |
<greg-g> |
nothing executing on deployment-bastion, that is |
[releng] |
| 16:38 |
<greg-g> |
same, nothing executing |
[releng] |
| 16:37 |
<greg-g> |
did that checklist once, jobs still not executing, doing again |
[releng] |
| 16:32 |
<greg-g> |
I'll start going through the checklist at https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update |
[releng] |
| 16:30 |
<hashar> |
deadlock on deployment-bastion slave. Someone need to restart Jenkins :( |
[releng] |