在scala中导入spark.implicits._ [英] Importing spark.implicits._ in scala

问题描述

我正在尝试导入spark.implicits._ 显然，这是scala中的一个类内的对象. 当我以类似方法导入它时:

I am trying to import spark.implicits._ Apparently, this is an object inside a class in scala. when i import it in a method like so:

def f() = {
  val spark = SparkSession()....
  import spark.implicits._
}

它可以正常工作，但是我正在编写一个测试类，并且我想使此导入可用于所有测试 我已经尝试过:

It works fine, however i am writing a test class and i want to make this import available for all tests I have tried:

class SomeSpec extends FlatSpec with BeforeAndAfter {
  var spark:SparkSession = _

  //This won't compile
  import spark.implicits._

  before {
    spark = SparkSession()....
    //This won't either
    import spark.implicits._
  }

  "a test" should "run" in {
    //Even this won't compile (although it already looks bad here)
    import spark.implicits._

    //This was the only way i could make it work
    val spark = this.spark
    import spark.implicits._
  }
}

这不仅看起来不好，我不想在每次测试时都这样做 这样做的正确"方法是什么?

Not only does this look bad, i don't want to do it for every test What is the "correct" way of doing it?

推荐答案

您可以执行类似于Spark测试套件中的操作.例如，这将起作用(受

You can do something similar to what is done in the Spark testing suites. For example this would work (inspired by SQLTestData):

class SomeSpec extends FlatSpec with BeforeAndAfter { self =>

  var spark: SparkSession = _

  private object testImplicits extends SQLImplicits {
    protected override def _sqlContext: SQLContext = self.spark.sqlContext
  }
  import testImplicits._

  before {
    spark = SparkSession.builder().master("local").getOrCreate()
  }

  "a test" should "run" in {
    // implicits are working
    val df = spark.sparkContext.parallelize(List(1,2,3)).toDF()
  }
}

或者，您可以使用

Alternatively you may use something like SharedSQLContext directly, which provides a testImplicits: SQLImplicits, i.e.:

class SomeSpec extends FlatSpec with SharedSQLContext {
  import testImplicits._

  // ...

}

这篇关于在scala中导入spark.implicits._的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！