3301-3350 of 4521 results (16ms)
2014-03-19 §
16:30 <hashar> Varnish caches in eqiad are failing puppet because there is no /dev/vdb. Will figure it out tomorrow :-] [releng]
16:15 <hashar> Applying role::logging::mediawiki::errors on deployment-fluoride.eqiad.wmflabs . It is not receiving anything yet though. [releng]
15:50 <hashar> fixed upd2log-mw daemon not starting on eqiad bastion ( /var/log/udp2log belonged to wrong UID/GID) [releng]
15:49 <hashar> deleted local user l10nupdate on deployment-bastion. It is in ldap now. [releng]
2014-03-17 §
15:02 <hashar> Starting copying /data/project from ptmpa to eqiad [releng]
14:46 <hashar> manually purging all commonswiki archived files (on beta of course) [releng]
2014-03-14 §
14:47 <hashar> changing uid/gid of mwdeploy which is now provisioned via LDAP (aka deleting local user and group on all instance + file permissions tweaks) [releng]
2014-03-11 §
10:46 <hashar> dropping some unused databases from deployment-sql instance. [releng]
2014-03-10 §
11:09 <hashar> Deleting http://simple.wikipedia.beta.wmflabs.org/wiki/MediaWiki:Robots.txt [releng]
09:54 <hashar> Reducing memcached instances to 3GB ( {{gerrit|115617}} ). Seems to fix writing to the EQIAD memcaches which only have 3GB [releng]
09:08 <hashar> Restarted bits cache (CPU / mem overload) [releng]
2014-03-06 §
09:07 <hashar> restarted varnish and varnish-frontend on deployment-cache-text1 [releng]
2014-03-05 §
17:26 <hashar> hacked in mwversioninuse to return "master=aawiki". Relaunched l10n job using mwdeploy user and then running mw-update-l10n [releng]
17:07 <hashar> mwversioninuse gives a wmf branch instead of master. That breaks l10n messages update and the job https://integration.wikimedia.org/ci/job/beta-code-update/ . Root cause is the python based scap. [releng]
2014-03-03 §
17:28 <manybubbles> doing an Elasticsearch reindex on beta before I try another one in production [releng]
2014-02-28 §
10:17 <hashar> Puppet running on varnish upload cache after several months. Might break random things in the process :( [releng]
2014-02-27 §
14:11 <manybubbles> upgrading beta to Elasticsearch 1.0 [releng]
2014-02-26 §
20:44 <hashar> Cleaning up commonswiki archived files with mwscript deleteArchivedFiles.php --wiki=commonswiki --delete [releng]
20:44 <hashar> deleted all files from http://commons.wikimedia.beta.wmflabs.org/wiki/Category:GWToolset_Batch_Upload (gwtoolset import test). Deleted File:Title_0* (Selenium tests). [releng]
15:06 <hashar> deleted all thumbs from shared directory: /data/project/upload7/*/*/thumb/* [releng]
14:54 <hashar> cleaning out 2013 archived logs. [releng]
2014-02-25 §
08:42 <hashar> Upgrading all varnishes. [releng]
2014-02-24 §
23:36 <MaxSem> Rolled back [releng]
23:25 <hoo> recursively chowned extensions/MobileFrontend to mwdeploy:mwdeploy [releng]
23:21 <hoo> chowned /data/project/apache/common-local/php-master/extensions/.git/modules/MobileFrontend/* to mwdeploy:mwdeploy [releng]
17:47 <MaxSem> Investigating a mobile bug, might cause intermittent problems [releng]
17:36 <MaxSem> Rebooted deployment-cache-mobile01 - was impossible to log into it though Varnish still worked [releng]
2014-02-21 §
19:42 <MaxSem> Adjusted read privs on /home/wikipedia/syslog/apache.log to allow fatalmonitor to work [releng]
2014-02-19 §
16:24 <hashar> -bastion : /etc/init.d/udp2log stop && /etc/init.d/udp2log-mw start (known bug) [releng]
16:23 <hashar> rebooting -bastion [releng]
16:22 <hashar> rebooting apache32 and apache33 breaking beta :-] [releng]
2014-02-17 §
15:26 <hashar> rebooting bits cache [releng]
2014-02-11 §
21:55 <manybubbles> update elasticsearch schema after recent changes. will run a links update as well [releng]
2014-02-06 §
22:20 <Krinkle> Manually ran changePassword.php to help someone (password reminder emails don't get sent) [releng]
14:43 <hashar> restarting udp2log-mw on deployment-bastion. Logstash.wmflabs.org no more receiving fatals logs since Jan 31st [releng]
2014-02-04 §
17:22 <hashar> fixed up beta-parsoid-update job so Parsoid should be up to date again. The issue is that the multigit job pointed to a wrong host (ZUUL_URL should be zuul.eqiad.wmnet) [releng]
13:33 <hashar> removing role::memcached from both apache servers [releng]
09:58 <hashar> rebooting all varnish caches [releng]
09:57 <hashar> Upgrading all varnish [releng]
2014-02-03 §
16:59 <hashar> upgrading varnish on deployment-parsoidcache3 [releng]
2014-01-30 §
19:35 <hashar> deployment-cache-bits03 restarted gmond, leaked memory. Upgrading varnish [releng]
19:32 <hashar> Canceled varnish package upgrade on deployment-cache-mobile01 , it runs a specific version ( 3.0.5plus~wmftest-wm1 ) instead of 3.0.3plus~rc1-wm29 [releng]
19:30 <hashar> upgrading varnish on deployment-cache-mobile01 [releng]
19:29 <hashar> upgrading varnish on deployment-cache-bits03 [releng]
19:29 <hashar> upgrading varnish on deployment-staging-cache-mobile02 [releng]
19:28 <hashar> upgrading varnish on deployment-cache-upload04 [releng]
19:27 <hashar> reenabling puppet on deployment-cache-mobile01 [releng]
17:10 <manybubbles> done reindexing beta. everything looks good [releng]
16:54 <manybubbles> reindexing beta like we're going to do in production when the release train departs later today [releng]
2014-01-28 §
17:10 <hashar> added addshore and jhall to project so they can grep logs [releng]