I'm implementing a template method pattern in Scala. The idea is that the method returns a Dataset[Metric]
.
But when I'm converting enrichedMetrics to a DataSet enrichedMetrics.as[Metric]
I have to use implicits in order to map the records to the specified type. This means passing a SparkSession to the MetricsProcessor
which seems not the best solution to me.
Is there a more proper way to implement the template method pattern in this case?
trait MetricsProcessor {
// Template method
def parseMetrics(startDate: Date, endDate: Date, metricId: Long): Dataset[Metric] = {
val metricsFromSource: DataFrame = queryMetrics(startDate, endDate)
val enrichedMetrics = enrichMetrics(metricsFromSource, metricId)
enrichedMetrics.as[Metric] <--- //requires spark.implicits
}
// abstract method
def queryMetrics(startDate: Date, endDate: Date): DataFrame
def enrichMetrics(metricsDf: DataFrame, metricId: Long): DataFrame = {
/*Default implementation*/
}
}
Aucun commentaire:
Enregistrer un commentaire