2201-2250 of 3795 results (11ms)
2014-08-15 §
21:42 <hashSpeleology> Beta cluster database updates are broken due to CentralNotice. Fix up is {{gerrit|154231}} [releng]
20:57 <hashSpeleology> deployment-rsync01 : deleting /usr/local/apache/common-local content. Then ln -s /srv/common-local /usr/local/apache/common-local as set by beta::common which is not applied on that host for some reason. {{bug|69590}} [releng]
20:55 <hashSpeleology> puppet administratively disabled on mediawiki02 . Assuming some work in progress on that host. Leaving it untouched [releng]
20:54 <hashSpeleology> puppet is proceeding on mediawiki01 [releng]
20:52 <hashSpeleology> attempting to unbreak mediawiki code update {{bug|69590}} by cherry picking {{gerrit|154329}} [releng]
20:39 <hashSpeleology> in case it is not in SAL. MediaWiki is no more synced to app server {{bug|69590}} [releng]
20:20 <hashSpeleology> rebooting mediawiki01 , /var refuses to clear out and stick at 100% usage [releng]
20:16 <hashSpeleology> cleaning up /var/log on deployment-mediawiki02 [releng]
20:14 <hashSpeleology> on deployment-mediawiki01 deleting /var/log/apache2/access.log.1 [releng]
20:13 <hashSpeleology> on deployment-mediawiki01 deleting /var/log/apache2/debug.log.1 [releng]
20:13 <hashSpeleology> bunch of instances have a full /var/log :-/ [releng]
11:37 <ori> deployment-cache-bits01 unresponsive; console shows OOMs: https://dpaste.de/LDRi/raw . rebooting [releng]
03:20 <jeremyb> 02:46:37 UTC <ebernhardson> !log beta /dev/vda1 full. moved /srv-old to /mnt/srv-old and freed up 2.1G [releng]
2014-08-14 §
12:23 <hashar> manually rebased operations/puppet.git on puppetmaster [releng]
2014-08-13 §
08:02 <hashar> beta-code-update-eqiad is running again [releng]
07:57 <hashar> fixing ownerships under /srv/scap-stage-dir/php-master/skins some files belong to root [releng]
07:55 <hashar> https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ is broken :-/ [releng]
2014-08-08 §
16:05 <bd808> Fixed merge conflict that was preventing updates on puppet master [releng]
2014-08-06 §
13:13 <hashar> https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ is running again [releng]
13:13 <hashar> removed a bunch of local hack on deployment-bastion:/srv/scap-stage-dir/php-master . That causes the git repo to be dirty and prevents scap from achieving git pull there [releng]
12:08 <hashar> Manually pruning whole text cache on deployment-cache-text02 [releng]
12:07 <hashar> Apache virtual hosts were not properly loaded on mediawiki02. I have hacked /etc/apache2/apache2.conf to make it Include Include /usr/local/apache/conf/all.conf (instead of main.conf which does not include everything) [releng]
08:43 <hashar> prunning cache on deployment-cache-text02 / restarting varnish [releng]
2014-08-02 §
08:53 <swtaarrs> rebuilt and restarted hhvm on deployment-mediawiki02 with potential fix [releng]
05:17 <swtaarrs> restarted hhvm on deployment-mediawiki0{1,2} to unwedge them [releng]
2014-08-01 §
15:03 <bd808> Updated cherry-pick of Iceb8f43 [releng]
15:02 <bd808> Cleaned up puppet repo on deployment-salt; merge conflicts with local Ia463120 hack; reapplied depool of deployment-mediawiki01 [releng]
14:50 <bd808> Restarted stuck hhvm on deployment-mediawiki02; apache had 89 children waiting for a response [releng]
13:27 <godog> changed inplace bt-hhvm on deployment-mediawiki01/02 to also copy the binary [releng]
05:32 <ori> depooled deployment-mediawiki02 to investigate HHVM lock-up by cherry-picking I7df8c5310 on beta. [releng]
00:40 <ori> disabled puppet on deployment-mediawiki{01,02} and enabled verbose apache logging [releng]
2014-07-31 §
22:41 <bd808> Restarted hhvm on -mediawiki{01,02}. Brett looked at 01 before I did and said "it's the same as before" [releng]
20:09 <cscott> updated OCG to version d2919c59eb09e09fc87777696411a070620aef45 [releng]
19:59 <hashar> Granted sudo right to cscott (under NDA). Will let him reboot OCG service [releng]
18:58 <ori> re-enabled puppet on deployment-mediawiki{01,02} [releng]
10:41 <hashar> Taking gdb traces of hhvm on mediawiki01 and mediawiki02. Restarting hhvm [releng]
05:08 <bd808> HHVM hung on both boxes. Grabbed core and backtrace before restarting [releng]
2014-07-30 §
19:59 <bd808> Created local commit 7d56b79 in puppet to work around bugs in Ia463120718dceab087ad3f8e3f35917fa879f387 [releng]
19:46 <bd808> Restored prior /etc/hhvm/php.ini from puppet filebucket archive on deployment-mediawiki0[12] [releng]
19:32 <bd808> Disabled puppet on deployment-mediawiki02 for the same reason [releng]
19:31 <bd808> Disabled puppet on deployment-mediawiki01; Ori will look into hhvm config changes that were being applied [releng]
16:52 <bd808> Fixed beta-scap-eqiad Jenkins job by correcting ssh problems in beta project [releng]
16:43 <bd808> Fixed ssh to jobrunner01 and videoscaler01 by correcting unrelated puppet manifest problem and forcing run via salt. [releng]
16:00 <bd808> Puppet runs on videoscaler01 and jobrunner01 failing for "Could not find dependency Ferm::Rule[bastion-ssh] for Ferm::Rule[deployment-bastion-scap-ssh]" [releng]
16:00 <bd808> Puppet seems manually disabled on apache0[12]. [releng]
15:59 <bd808> Can't ssh to apache0[12], videoscaler01 and jobrunner01. Puppet not running on any of them. libnss-ldapd unattended update has broken /etc/nslcd.conf [releng]
15:23 <bd808> Removed cherry-pick for Iac547efa83cf059a1276b6e279c3ebd4c7224b2c and updated cherry-pick for I5afba2c6b0fbf90ff8495cc4a82f5c7851893b52 to latest patch set. [releng]
15:05 <bd808> Two cherry-picks in puppet conflicting with merged production changes: I5afba2c6b0fbf90ff8495cc4a82f5c7851893b52 and Iac547efa83cf059a1276b6e279c3ebd4c7224b2c (ori, twentyafterfour) [releng]
14:49 <bd808> Started apache2 service on deployment-mediawiki01 [releng]
14:16 <hashar> rebooting hhvm [releng]