如何在ScalaTest测试中正确使用Spark? [英] How to correctly use Spark in ScalaTest tests?
问题描述
我有多个ScalaTest类,它们使用 BeforeAndAfterAll
构造一个 SparkContext
并随后将其停止,如下所示:
class MyTest使用BeforeAndAfterAll {私人var sc:SparkContext = null覆盖受保护的def beforeAll():单位= {sc = ...//创建SparkContext}覆盖保护的def afterAll():单位= {sc.stop()}//我的测试如下}
这些测试从IntelliJ IDEA启动时运行良好,但是运行 sbt测试
时,我得到 WARN SparkContext:正在构造另一个SparkContext(或在其构造函数中引发了异常).这可能表示一个错误,因为在此JVM中只能运行一个SparkContext(请参阅SPARK-2243).
,然后,我认为还有许多其他与此问题相关的异常.>
如何正确使用Spark?我是否必须为整个测试套件创建一个全局 SparkContext
,如果是,该怎么做?
似乎我看不见树木的树木,我在 build.sbt
中忘记了以下内容:>测试中的
parallelExecution:=否
在此行中,测试运行.
I have multiple ScalaTest classes which use BeforeAndAfterAll
to construct a SparkContext
and stop it afterwards like so:
class MyTest extends FlatSpec with Matchers with BeforeAndAfterAll {
private var sc: SparkContext = null
override protected def beforeAll(): Unit = {
sc = ... // Create SparkContext
}
override protected def afterAll(): Unit = {
sc.stop()
}
// my tests follow
}
These tests run fine when started from IntelliJ IDEA, but when running sbt test
, I get WARN SparkContext: Another SparkContext is being constructed (or threw an exception in its constructor). This may indicate an error, since only one SparkContext may be running in this JVM (see SPARK-2243).
, and after that, a bunch of other exceptions which are, I suppose, related to this issue.
How to correctly use Spark? Do I have to create one global SparkContext
for the whole test suite, and if yes, how do I do this?
Seems like I lost the sight of the wood for the trees, I forgot the following line in my build.sbt
:
parallelExecution in test := false
With this line, the test runs.
这篇关于如何在ScalaTest测试中正确使用Spark?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!