I am looking for a good way to insure that my app is only using one single Spark Context (sc). While developing I often run into errors and have to restart my Play! server to re test my modifications. Would a Singleton pattern be solution ?
object sparckContextSingleton {
@transient private var instance: SparkContext = _
private val conf : SparkConf = new SparkConf()
.setMaster("local[2]")
.setAppName("myApp")
def getInstance(): SparkContext = {
if (instance == null){
instance = new SparkContext(conf)
}
instance
}
}
This does not make a good job. Should I stop the SparkContext?
Aucun commentaire:
Enregistrer un commentaire