mercredi 22 juillet 2015

Good Practice to insure only one Spark context in application

I am looking for a good way to insure that my app is only using one single Spark Context (sc). While developing I often run into errors and have to restart my Play! server to re test my modifications. Would a Singleton pattern be solution ?

object sparckContextSingleton {
  @transient private var instance: SparkContext = _
  private val conf : SparkConf = new SparkConf()
    .setMaster("local[2]")
    .setAppName("myApp")

  def getInstance(): SparkContext = {
    if (instance == null){
      instance = new SparkContext(conf)
    }
    instance
  }  
}

This does not make a good job. Should I stop the SparkContext?

Aucun commentaire:

Enregistrer un commentaire