501-550 of 1833 results (20ms)
2018-05-08 §
20:12 <milimetric> aborting deployment, will deploy data truncation script too [analytics]
20:08 <milimetric> deploying refinery to relaunch geoeditors job [analytics]
17:57 <joal> Mvoe recomputed 2018-03 history snapshot in place of old one (T194075) [analytics]
15:38 <joal> Try again (last time) to rerun mediawiki-history-druid-wf-2018-04 [analytics]
15:06 <ottomata> beginnng Kafka upgrade of main-codfw: T167039 [analytics]
08:01 <elukey> removed cassandra-metrics-collector (graphite) from aqs nodes [analytics]
07:42 <joal> Rerun mediawiki-history-druid-wf-2018-04 in a non-sync way with mediawiki-reduced [analytics]
06:41 <elukey> rolling restart of druid-historicals on druid100[456] due to half of the segments not avaiable [analytics]
2018-05-07 §
12:05 <joal> Rerun mediawiki-history-reduced-wf-2018-04 [analytics]
09:18 <elukey> re-run webrequest-load-wf-text-2018-5-7-7 - failed due to reimages [analytics]
2018-05-04 §
10:11 <elukey> d-[123] Druid cluster upgraded to 0.11 in labs (project analytics) [analytics]
2018-05-03 §
20:29 <milimetric> fixed wikimetrics issues, working fine again [analytics]
19:19 <milimetric> wikimetrics is partly broken until I can figure out what’s going on [analytics]
2018-05-02 §
17:33 <joal> Rerun webrequest-load-wf-text-2018-5-2-15 [analytics]
16:41 <joal> Manually silence pageview-whitelist alarm overwriting /wmf/refinery/current/static_data/pageview/whitelist/whitelist.tsv [analytics]
16:27 <joal> 2018-05-02T14 webrequest dataloss warnings have been checked and are false positives [analytics]
16:17 <joal> Restart oozie mediawiki-history-denormalize job after deploy [analytics]
16:14 <ottomata> bounced eventlogging-consumer@mysql-m4-master-00 after kafka jumbo 1.1.0 upgrade [analytics]
16:05 <joal> Restart oozie webrequest bundle after deploy [analytics]
15:20 <joal> Deploying refinery to hadoop [analytics]
14:45 <joal> Deploying refinery using Scap [analytics]
14:16 <joal> Refinery-source version 0.0.63 finally released to Archiva! [analytics]
13:49 <ottomata> beginning upgrade of kafka-jumbo brokers from 1.0.0 -> 1.1.0 : T193495 [analytics]
13:20 <elukey> restart druid broker on druid100[1-3] to enable druid.sql.enable: true [analytics]
2018-05-01 §
15:33 <elukey> restart historical on druid1003 - exceptions in the logs [analytics]
15:22 <elukey> restart druid-historical on druid1002 - Caused by: java.lang.IllegalArgumentException: Could not resolve type id 'hdfs' into a subtype of [analytics]
11:44 <joal> False positive only in webrequest-load-check_sequence_statistics-wf-upload-2018-5-1-6 [analytics]
07:14 <joal> Rerun webrequest-druid-daily-wf-2018-4-30 [analytics]
06:24 <elukey> roll restart of all middlemanagers on druid100[123] - realtime tasks piled up from hours [analytics]
2018-04-30 §
23:04 <ottomata> blacklisting change-prop and job queue topics from main-eqiad -> analytics (eqiad) [analytics]
22:55 <ottomata> bouncing kafka main-eqiad -> eqiad (analytics) mirror maker [analytics]
19:34 <joal> Retry releasing refinery-source to archiva [analytics]
18:43 <joal> Releasing refinery-source [analytics]
15:53 <joal> Resume webrequest-druid-hourly-coord and pageview-druid-hourly-coord [analytics]
14:23 <joal> Suspend webrequest-druid-hourly-coord and pageview-druid-hourly-coord before druid upgrade [analytics]
14:23 <elukey> disabled cron/check on analytics1003 to respawn banner impressions if needed [analytics]
14:21 <joal> Kill BannerImpressionStream job before upgrading druid [analytics]
2018-04-25 §
14:39 <elukey> re-enable camus after maintenance [analytics]
14:37 <elukey> restart hive-server2 on analytics1003 to pick up settings in https://gerrit.wikimedia.org/r/428919 [analytics]
13:40 <elukey> stop camus on an1003 as prep step to gracefully restart hive server [analytics]
12:24 <joal> Only false positive for Data Loss Warning - Workflow webrequest-load-check_sequence_statistics-wf-upload-2018-4-25-10 [analytics]
2018-04-24 §
16:30 <elukey> restart hadoop hdfs journalnode on analytics1035/52 to pick up prometheus jmx settings [analytics]
14:41 <elukey> restart hadoop hdfs journalnode on analytics1028 to pick up jmx settings [analytics]
12:08 <elukey> restart webrequest-load-wf-text-2018-4-24-9 via Hue (failed due to reimages) [analytics]
06:57 <joal> correct reindextion job: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0033859-180330093100664-oozie-oozi-C/ [analytics]
06:55 <joal> Reindextion job: https://hue.wikimedia.org/oozie/list_oozie_coordinator/0033855-180330093100664-oozie-oozi-C/ [analytics]
06:54 <joal> Manually reindexing all of mediawiki-history for snapshot 2018-03 after having messed it with job testing [analytics]
2018-04-23 §
20:41 <milimetric> deployed a version of wikistats with all but reading metrics disabled to stop showing bad data [analytics]
19:34 <elukey> deploy https://gerrit.wikimedia.org/r/428331 for Pivot [analytics]
14:10 <ottomata> switching main -> analytics MirrorMaker to --new.consumer (temporarily stopping puppet on kafka101[234]) https://phabricator.wikimedia.org/T192387 [analytics]