mardi 23 juillet 2019

Scala - template method pattern in a trait

I'm implementing a template method pattern in Scala. The idea is that the method returns a Dataset[Metric].

But when I'm converting enrichedMetrics to a DataSet enrichedMetrics.as[Metric] I have to use implicits in order to map the records to the specified type. This means passing a SparkSession to the MetricsProcessor which seems not the best solution to me.

Is there a more proper way to implement the template method pattern in this case?

  trait MetricsProcessor  {

  // Template method
  def parseMetrics(startDate: Date, endDate: Date, metricId: Long): Dataset[Metric] = {
    val metricsFromSource: DataFrame = queryMetrics(startDate, endDate)
    val enrichedMetrics = enrichMetrics(metricsFromSource, metricId)
    enrichedMetrics.as[Metric] <--- //requires spark.implicits
  }

  // abstract method
  def queryMetrics(startDate: Date, endDate: Date): DataFrame

  def enrichMetrics(metricsDf: DataFrame, metricId: Long): DataFrame = {
  /*Default implementation*/
  }
 }

Aucun commentaire:

Enregistrer un commentaire