|
2015-04-07
§
|
| 12:11 |
<hashar> |
retriggering a bunch of browser tests hitting beta.wmflabs.org |
[releng] |
| 12:07 |
<hashar> |
Puppet being fixed, it is finishing the installation of integration-slave-trusty-*** hosts |
[releng] |
| 12:03 |
<hashar> |
Browser tests against beta cluster were all failing due to an improper DNS resolver being applied on CI labs instances {{bug|T95273}}. Should be fixed now. |
[releng] |
| 12:00 |
<hashar> |
running puppet on all integration machines and resigning puppet client certs |
[releng] |
| 11:31 |
<hashar> |
integration-puppetmaster is back and operational with local puppet client working properly. |
[releng] |
| 11:28 |
<hashar> |
restored /etc/puppet/fileserver.conf |
[releng] |
| 11:08 |
<hashar> |
dishing out puppet SSL configuration on all integratio nodes. Can't figure out so lets restart from scratch |
[releng] |
| 10:52 |
<hashar> |
made puppetmaster certname = integration-puppetmaster.eqiad.wmflabs instead of the ec2 id :( |
[releng] |
| 10:49 |
<hashar> |
manually hacking integration-puppetmaster /etc/puppet/puppet.conf config file which is missing the [master] section |
[releng] |
| 01:25 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/202300 |
[releng] |
|
2015-04-03
§
|
| 23:47 |
<greg-g> |
for Krinkle 23:31 "Finished npm upgrade on trusty slaves." |
[releng] |
| 23:08 |
<Krinkle> |
Finished npm upgrade on precise slaves. Rolling on trusty slaves now. |
[releng] |
| 22:55 |
<bd808> |
Updated scap to a1a5235 (Add a logo banner to scap) |
[releng] |
| 21:31 |
<Krinkle> |
Upgrading npm from v2.4.1 to v2.7.5 (rolling, slave by slave graceful) |
[releng] |
| 21:11 |
<^d> |
puppet re-enabled on staging-palladium, running fine again |
[releng] |
| 21:05 |
<Krinkle> |
Delete unfinished/unpoooled instances integration-slave-precise-1011-1014. (T94916) |
[releng] |
| 14:49 |
<hashar> |
integration-slave-jessie-1001 : manually installed jenkins-debian-glue Debian packages. It is pending upload by ops to apt.wikimedia.org {{bug|T95006}} |
[releng] |
| 12:56 |
<hashar> |
installed zuul_2.0.0-304-g685ca22-wmf1precise1_amd64.deb on integration-slave-precise-101* instances |
[releng] |
| 12:56 |
<hashar> |
installed zuul_2.0.0-304-g685ca22-wmf1precise1_amd64.deb on integration-slave-precise-1011.eqiad.wmflabs |
[releng] |
| 12:35 |
<hashar> |
Switching Jessie slave from role::ci::slave::labs::common to role::ci::slave::labs which will bring a whole lot of packages and break |
[releng] |
| 12:28 |
<hashar> |
integration-slave-jessie-1001 applying role::ci::slave::labs::common to pool it as a very basic Jenkins slave |
[releng] |
| 12:19 |
<hashar> |
enabled puppetmaster::autosigner on integration-puppetmaster |
[releng] |
| 11:58 |
<hashar> |
Applied role::ci::slave::labs on integration-slave-precise-101[1-4] that Timo created earlier |
[releng] |
| 11:58 |
<hashar> |
Cherry picked a couple patches to fix puppet Package[] definitions issues |
[releng] |
| 11:49 |
<hashar> |
made integration-puppetmaster to self update its puppet clone |
[releng] |
| 11:42 |
<hashar> |
recreating integration-slave-precise-1011 stalled with a puppet oddity related to Package['gdb'] defined twice {{bug|T94917}} |
[releng] |
| 11:30 |
<hashar> |
integration-puppetmaster migrated down to Precise |
[releng] |
| 11:23 |
<hashar> |
rebooting integration-publisher : cant ssh to it |
[releng] |
| 10:37 |
<hashar> |
disabled some hiera configuration related to puppetmaster. |
[releng] |
| 10:22 |
<hashar> |
Created instance i-00000a4a with image "ubuntu-12.04-precise" and hostname i-00000a4a.eqiad.wmflabs. |
[releng] |
| 10:21 |
<hashar> |
downgrading integration-puppetmaster from Trusty to Precise https://phabricator.wikimedia.org/T94927 |
[releng] |
| 05:42 |
<legoktm> |
deploying https://gerrit.wikimedia.org/r/200744 |
[releng] |
| 03:58 |
<Krinkle> |
Jobs were throwing NOT_RECOGNISED. Relaunched Gearman. Jobs are now happy again. |
[releng] |
| 03:51 |
<Krinkle> |
Jenkins is unable to re-establish Gearman connection. Have to force restart Jenkins master. |
[releng] |
| 03:44 |
<greg-g> |
*unable |
[releng] |
| 03:44 |
<Krinkle> |
References to past hour of builds have been restored. But Jenkins is still enable to make new references properly. New builds are 404'ing the same way. |
[releng] |
| 03:42 |
<Krinkle> |
Reloading Jenking config repaired the broken references. Build urls are now resolving again. |
[releng] |
| 03:26 |
<Krinkle> |
Reloading Jenkins configuration from disk to mitigate |
[releng] |
| 03:18 |
<Krinkle> |
The failure started at 03:03 exactly. The newer build metadata exists at /var/lib/jenkins/jobs/:jobname/builds/:nr, but the jobs/*/last*Build symlinks are no longer updated. |
[releng] |
| 02:47 |
<Krinkle> |
Reloading Zuul to deploy https://gerrit.wikimedia.org/r/201644 |
[releng] |