| 2024-01-31
      
      § | 
    
  | 19:59 | <joal> | Deploying refinery with scap for second hotfix | [analytics] | 
            
  | 19:14 | <joal> | Backfill wmf_traffic.aqs_hourly | [analytics] | 
            
  | 19:14 | <joal> | Drop/Recreate wmf_traffic.aqs_hourly table (iceberg) to change compression format | [analytics] | 
            
  | 18:40 | <phuedx> | phuedx@deploy2002 Finished deploy [airflow-dags/analytics@5078a6b]: (no justification provided) (duration: 00m 28s) | [analytics] | 
            
  | 18:40 | <phuedx> | phuedx@deploy2002 Started deploy [airflow-dags/analytics@5078a6b]: (no justification provided) | [analytics] | 
            
  | 17:46 | <phuedx> | Deployed refinery using scap, then deployed onto hdfs | [analytics] | 
            
  | 17:40 | <joal> | pause pageview_actor_hourly for deploy | [analytics] | 
            
  | 17:35 | <phuedx> | phuedx@deploy2002 Finished deploy [analytics/refinery@bef134c] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bef134c2] (duration: 03m 29s) | [analytics] | 
            
  | 17:31 | <phuedx> | phuedx@deploy2002 Started deploy [analytics/refinery@bef134c] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@bef134c2] | [analytics] | 
            
  | 17:31 | <phuedx> | phuedx@deploy2002 Finished deploy [analytics/refinery@bef134c] (thin): Regular analytics weekly train THIN [analytics/refinery@bef134c2] (duration: 00m 08s) | [analytics] | 
            
  | 17:31 | <phuedx> | phuedx@deploy2002 Started deploy [analytics/refinery@bef134c] (thin): Regular analytics weekly train THIN [analytics/refinery@bef134c2] | [analytics] | 
            
  | 17:30 | <phuedx> | phuedx@deploy2002 Finished deploy [analytics/refinery@bef134c]: Regular analytics weekly train [analytics/refinery@bef134c2] (duration: 11m 05s) | [analytics] | 
            
  | 17:19 | <phuedx> | phuedx@deploy2002 Started deploy [analytics/refinery@bef134c]: Regular analytics weekly train [analytics/refinery@bef134c2] | [analytics] | 
            
  | 17:02 | <phuedx> | phuedx@deploy2002 Finished deploy [analytics/refinery@2c00cad] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2c00cad1] (duration: 03m 35s) | [analytics] | 
            
  | 17:00 | <phuedx> | phuedx@deploy2002 Started deploy [analytics/refinery@2c00cad] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@2c00cad1] | [analytics] | 
            
  | 16:57 | <phuedx> | phuedx@deploy2002 Finished deploy [analytics/refinery@2c00cad] (thin): Regular analytics weekly train THIN [analytics/refinery@2c00cad1] (duration: 00m 06s) | [analytics] | 
            
  | 16:57 | <phuedx> | phuedx@deploy2002 Started deploy [analytics/refinery@2c00cad] (thin): Regular analytics weekly train THIN [analytics/refinery@2c00cad1] | [analytics] | 
            
  | 16:53 | <phuedx> | phuedx@deploy2002 Finished deploy [analytics/refinery@2c00cad]: Regular analytics weekly train [analytics/refinery@2c00cad1] (duration: 09m 52s) | [analytics] | 
            
  | 16:52 | <phuedx> | Regular analytics weekly train [analytics/refinery@$(git rev-parse --short HEAD)] | [analytics] | 
            
  | 12:12 | <btullis> | rebooting dbstore1009 for new kernel version (T356239) | [analytics] | 
            
  | 11:56 | <btullis> | rebooting dbstore1008 for new kernel version (T356239) | [analytics] | 
            
  | 10:57 | <btullis> | deploying https://gerrit.wikimedia.org/r/c/analytics/superset/deploy/+/994213 to superset-next to test nested display of presto columns | [analytics] | 
            
  
    | 2024-01-15
      
      § | 
    
  | 17:02 | <btullis> | roll-restarting public druid cluster | [analytics] | 
            
  | 17:01 | <btullis> | roll-restarting analytics druid cluster | [analytics] | 
            
  | 16:55 | <joal> | Clearing analytics failed aiflow tasks after fix | [analytics] | 
            
  | 16:47 | <btullis> | restarted the hive-server2 and hive-metastore services on an-coord100[3-4] which had been accidentally omitted earlier for T332573 | [analytics] | 
            
  | 12:00 | <btullis> | removing all downtime for hadoop-all for T332573 | [analytics] | 
            
  | 11:57 | <btullis> | un-pausing all previously paused DAGS on all airflow instances for T332573 | [analytics] | 
            
  | 11:55 | <btullis> | re-enabling gobblin jobs | [analytics] | 
            
  | 11:38 | <brouberol> | redeploying the Spark History Server to pick up the new HDFS namenodes - T332573 | [analytics] | 
            
  | 11:29 | <btullis> | puppet runs cleanly on an-master1003 and it is the active namenode - running puppet an an-master1004. | [analytics] | 
            
  | 11:20 | <btullis> | running puppet on an-master1003 to set it to active for T332573 | [analytics] | 
            
  | 11:16 | <btullis> | running puppet on journal nodes first for T332573 | [analytics] | 
            
  | 11:03 | <btullis> | stopping all hadoop services | [analytics] |