9251-9300 of 10000 results (39ms)
2016-05-24 §
07:03 <mobrovac> rebooting deployment-tin, can't log in [releng]
2016-05-23 §
19:35 <hashar> killed all mysqld process on Trusty CI slaves [releng]
15:49 <thcipriani> beta code update not running, disconnect-reconnect dance resulted in: [05/23/16 15:48:39] [SSH] Authentication failed. [releng]
14:32 <jzerebecki> offlined integration-slave-trusty-1004 because it can't connect to mysql T135997 [releng]
13:32 <hashar> Upgrading Jenkins git plugins and restarting Jenkins [releng]
11:01 <hashar> Upgrading hhvm on Trusty slaves. Bring him hhvm compiled against libicu52 instead of libicu48 [releng]
09:12 <_joe_> deployment-prep: all hhvm hosts in beta upgraded to run on the newer libicu; now running updateCollation.php (T86096) [releng]
09:11 <hashar> Image ci-jessie-wikimedia-1463994307 in wmflabs-eqiad is ready [releng]
09:01 <hashar> Image ci-trusty-wikimedia-1463993508 in wmflabs-eqiad is ready [releng]
08:56 <_joe_> deployment-prep: starting upgrade of HHVM to a version linked to libicu52, T86096 [releng]
08:54 <hashar> Regenerating Nodepool image manually. Broke over the week-end due to a hhvm/libicu transition. Should get pip 8.1.x now [releng]
2016-05-20 §
20:30 <bd808> Killing https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/43608/ which has been running for 5 hours [releng]
2016-05-19 §
16:47 <thcipriani> deployment-tin jenkins worker seems to be back online after [https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code.2Fdb_update some prodding] [releng]
16:41 <thcipriani> beta-code-update eqiad hung for past few hours [releng]
15:16 <hashar> Restarted zuul-merger daemons on both gallium and scandium : file descriptors leaked [releng]
11:59 <hashar> CI: salt -v '*' cmd.run 'pip install --upgrade pip==8.1.2' [releng]
11:54 <hashar> Upgrading pip on CI slaves from 7.0.1 to 8.1.2 https://gerrit.wikimedia.org/r/#/c/289639/ [releng]
10:15 <hashar> puppet broken on deployment-tin : ?[1;31mError: Could not retrieve catalog from remote server: Error 400 on SERVER: Invalid parameter trusted_group on node deployment-tin.deployment-prep.eqiad.wmflabs?[0m [releng]
2016-05-18 §
13:16 <Amir1> deploying a05e830 to ores nodes (sca01 and ores-web) [releng]
12:46 <urandom> (re)cherry-picking c/284078 to deployment-prep [releng]
11:36 <hashar> Restarted qa-morebots [releng]
11:36 <hashar> Marked mediawiki/core/vendor repository has hidden in Gerrit. It got moved to mediawiki/vendor including the whole history Settings page: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/core/vendor [releng]
11:29 <hashar> Marked mediawiki/core/vendor repository has hidden in Gerrit. It got moved to mediawiki/vendor including the whole history Settings page: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/core/vendor [releng]
2016-05-17 §
16:01 <twentyafterfour> branching wmf/1.28.0-wmf.2 [releng]
15:01 <hashar> beta: salt -v '*' cmd.run 'groupdel mwdeploy' [releng]
01:01 <thcipriani> nodepool CI was down, slowly recovering [releng]
2016-05-13 §
14:39 <thcipriani> remove shadow l10nupdate user from deployment-tin and mira in beta [releng]
10:20 <hashar> Put integration-slave-trusty-1004 offline. Ssh/passwd is borked T135217 [releng]
09:59 <hashar> Deleting non nodepool mediawiki PHPUnit jobs for T135001 (mediawiki-phpunit-hhvm mediawiki-phpunit-parsertests-hhvm mediawiki-phpunit-parsertests-php55 mediawiki-phpunit-php55) [releng]
04:06 <thcipriani|afk> changed ownership of mwdeploy public keys post shadow mwdeploy user removal is important [releng]
03:47 <thcipriani|afk> ldap failure has created a shadow mwdeploy user on beta, deleted using vipw [releng]
2016-05-12 §
22:53 <bd808> Started dead mysql on integration-slave-precise-1011 [releng]
2016-05-11 §
21:05 <hashar> Reloading Zuul to deploy https://gerrit.wikimedia.org/r/288128 #T134946 [releng]
20:26 <hashar> rebooting integration-slave-trusty-1016 is back up [releng]
20:15 <hashar> rebooting integration-slave-trusty-1016 unreachable somehow [releng]
16:43 <hashar> Reduced number of executors on Trusty instances from 3 to 2. Memory get exhausted causing the tmpfs to drop files and thus MW jobs to fail randomly. [releng]
13:33 <hashar> Added contint::packages::php to Nodepool images T119139 [releng]
12:59 <hashar> Dropping texlive and its dependencies from gallium. [releng]
12:52 <hashar> deleted integration-dev [releng]
12:51 <hashar> creating integration-dev instance to hopefully have Shinken clean itself [releng]
11:42 <hashar> rebooting deployment-aqs01 via wikitech T134981 [releng]
10:46 <hashar> beta/ci puppetmaster : deleting old tags in /var/lib/git/operations/puppet and repacking the repos [releng]
08:49 <hashar> Deleting instances deployment-memc02 and deployment-memc03 (Precise instances, migrated to Jessie) #T134974 [releng]
08:43 <hashar> Beta: switching memcached to new Jessie servers by cherry picking https://gerrit.wikimedia.org/r/#/c/288156/ and running puppet on mw app servers #T134974 [releng]
08:20 <hashar> Creating deployment-memc04 and deployment-memc05 to switch beta cluster memcached to Jessie. m1.medium with security policy "cache" T13497 [releng]
01:44 <matt_flaschen> Created Flow-specific External Store tables (blobs_flow1) on all wiki databases on Beta Cluster: T128417 [releng]
2016-05-10 §
19:17 <hashar> beta / CI purging old Linux kernels: salt -v '*' cmd.run 'dpkg -l|grep ^rc|awk "{ print \$2 }"|grep linux-image|xargs dpkg --purge' [releng]
17:34 <cscott> updated OCG to version b0c57a1c6890e9fa1f2c3743fc14cb6a7f244fc3 [releng]
16:44 <bd808> Cleaned up 8.5G of pbuilder tmp output on integration-slave-jessie-1001 with `sudo find /mnt/pbuilder/build -maxdepth 1 -type d -mtime +1 -exec rm -r {} \+` [releng]
16:35 <bd808> https://integration.wikimedia.org/ci/job/debian-glue failure on integration-slave-jessie-1001 due to /mnt being 100$ full [releng]