4701-4750 of 5881 results (28ms)
2018-03-12 §
18:17 <ottomata> bouncing kafka mm eqiad -> jumbo witih acks=1 [analytics]
18:10 <ottomata> bouncing kafka mirrormaker for main-eqiad -> jumbo with buffer.memory=128M [analytics]
17:34 <joal> Restart mediawiki-history-reduced oozie job to add a dependency [analytics]
16:55 <joal> Restart mobile_apps_session_metrics [analytics]
16:52 <joal> Deploying refinery on HDFS for mobile_apps patch [analytics]
16:26 <joal> Deploying refinery again to provide patch for mobile_apps_session_metric job [analytics]
15:09 <joal> Deploy refinery onto hdfs [analytics]
15:07 <joal> Deploy refinery from scap [analytics]
14:32 <elukey> restart druid-broker on druid1004 - no /var/log/druid/broker.log after 2018-03-10T22:38:52 (java.io.IOException: Too many open files_ [analytics]
08:50 <elukey> fixed evenglog1002's ipv6 (https://gerrit.wikimedia.org/r/#/c/418714/) [analytics]
2018-03-10 §
09:07 <joal> Rerun clickstream-wf-2018-2 [analytics]
00:32 <milimetric> finished sqooping pagelinks for missing dbs, hdfs -put a SUCCESS flag in the 2018-02 snapshot, jobs should run unless Hue is still lying to itself [analytics]
2018-03-09 §
17:29 <joal> Rerun mediawiki-history-reduced job after having manually repaired wmf_raw.mediawiki_project_namespace_map [analytics]
2018-03-08 §
18:05 <ottomata> bouncing ResourceManagers [analytics]
08:54 <elukey> re-enable camus after reboots [analytics]
07:15 <elukey> disable Camus on an1003 to allow the cluster to drain - prep step for an100[123] reboot [analytics]
2018-03-07 §
07:15 <elukey> manually re-run wikidata-articleplaceholder_metrics-wf-2018-3-6 [analytics]
2018-03-06 §
20:44 <ottomata> reverted change to point mediawiki monolog kafka producers at kafka jumbo-eqiad until deployment train is done T188136 [analytics]
20:35 <ottomata> pointing mediawiki monolog kafka producers at kafka jumbo-eqiad cluster: T188136 [analytics]
19:06 <elukey> cleaned up id=0 rows on db1108 (log database) for T188991 [analytics]
10:19 <elukey> restart webrequest-load-wf-upload-2018-3-6-7 (failed due to reboots) [analytics]
10:08 <elukey> re-starting mysql consumers on eventlog1001 [analytics]
09:41 <elukey> stop eventlogging's mysql consumers for db1107 (el master) kernel updates [analytics]
2018-03-05 §
18:22 <elukey> restart webrequest-load-wf-upload-2018-3-5-16 via Hue (failed due to reboots) [analytics]
18:21 <elukey> restart webrequest-load-wf-text-2018-3-5-16 via Hue (failed due to reboots) [analytics]
15:00 <mforns> rerun mediacounts-load-wf-2018-3-5-9 [analytics]
10:57 <joal> Relaunch Mediawiki-history job manually from spark2 to see if new versions helps [analytics]
10:57 <joal> Killing failing Mediawiki-History job for 2018-03 [analytics]
2018-03-02 §
15:33 <mforns> rerun webrequest-load-wf-text-2018-3-2-12 [analytics]
2018-03-01 §
14:59 <elukey> shutdown deployment-eventlog02 in favor of deployment-eventlog05 in deployment-prep (Ubuntu -> Debian EL migration) [analytics]
09:45 <elukey> rerun webrequest-load-wf-text-2018-3-1-6 manually, failed due to analytics1030's reboot [analytics]
2018-02-28 §
22:09 <milimetric> re-deployed refinery for a small docs fix in the sqoop script [analytics]
17:55 <milimetric> Refinery synced to HDFS, deploy completed [analytics]
17:40 <milimetric> deploying Refinery [analytics]
08:38 <joal> rerun cassandra-hourly-wf-local_group_default_T_pageviews_per_project_v2-2018-2-27-15 [analytics]
2018-02-27 §
19:12 <ottomata> updating spark2-* CLIs to spark 2.2.1: T185581 [analytics]
2018-02-21 §
20:48 <ottomata> now running 2 camus webrequest jobs, one consuming from jumbo (no data yet), the other from analytics. these should be fine to run in parallel. [analytics]
07:21 <elukey> reboot db1108 (analytics-slave.eqiad.wmnet) for mariadb+kernel updates [analytics]
2018-02-19 §
17:14 <elukey> deployed eventlogging - https://gerrit.wikimedia.org/r/#/c/405687/ [analytics]
07:35 <elukey> re-run wikidata-specialentitydata_metrics-wf-2018-2-17 via Hue [analytics]
2018-02-16 §
15:41 <elukey> add analytics1057 back in the Hadoop worker pool after disk swap [analytics]
10:55 <elukey> increased topic partitions for netflow to 3 [analytics]
2018-02-15 §
13:54 <milimetric> deployment of refinery and refinery-source done [analytics]
12:52 <joal> Killing webrequest-load bundle (next restart should be at hour 12:00) [analytics]
08:18 <elukey> removed jmxtrans and java 7 from analytics1003 and re-launched refinery-drop-mediawiki-snapshots [analytics]
07:51 <elukey> removed default-java packages from analytics1003 and re-launched refinery-drop-mediawiki-snapshots [analytics]
2018-02-14 §
13:44 <elukey> rollback java 8 upgrade for archiva - issues with Analytics builds [analytics]
13:35 <elukey> installed openjdk-8 on meitnerium, manually upgraded java-update-alternatives to java8, restarted archiva [analytics]
13:14 <elukey> removed java 7 packages from analytics100[12] [analytics]
12:43 <elukey> jmxtrans removed from all the Hadoop workers [analytics]