无法在 Spark-2.2.0 - Scala-2.11.8 上运行单元测试(scalatest) [英] Unable to run Unit tests (scalatest) on Spark-2.2.0 - Scala-2.11.8

问题描述

无法在 spark-2.2.0 上运行带有上下文的 scalatest

Unable to run run scalatest with context on spark-2.2.0

堆栈跟踪:

异常或错误导致运行中止:org.apache.spark.sql.test.SharedSQLContext.eventually(Lorg/scalatest/concurrent/PatienceConfiguration$Timeout;Lscala/Function0;Lorg/scalatest/concurrent/AbstractPatienceConfiguration$PatienceConfig;)Ljava/lang/Object;java.lang.NoSuchMethodError: org.apache.spark.sql.test.SharedSQLContext.eventually(Lorg/scalatest/concurrent/PatienceConfiguration$Timeout;Lscala/Function0;Lorg/scalatest/concurrent/AbstractPatienceConfiguration$PatienceConfig;)Ljava/lang/Object;在 org.apache.spark.sql.test.SharedSQLContext$class.afterEach(SharedSQLContext.scala:92)在 testUtils.ScalaTestWithContext1.afterEach(ScalaTestWithContext1.scala:7)在 org.scalatest.BeforeAndAfterEach$$anonfun$1.apply$mcV$sp(BeforeAndAfterEach.scala:234)

An exception or error caused a run to abort: org.apache.spark.sql.test.SharedSQLContext.eventually(Lorg/scalatest/concurrent/PatienceConfiguration$Timeout;Lscala/Function0;Lorg/scalatest/concurrent/AbstractPatienceConfiguration$PatienceConfig;)Ljava/lang/Object; java.lang.NoSuchMethodError: org.apache.spark.sql.test.SharedSQLContext.eventually(Lorg/scalatest/concurrent/PatienceConfiguration$Timeout;Lscala/Function0;Lorg/scalatest/concurrent/AbstractPatienceConfiguration$PatienceConfig;)Ljava/lang/Object; at org.apache.spark.sql.test.SharedSQLContext$class.afterEach(SharedSQLContext.scala:92) at testUtils.ScalaTestWithContext1.afterEach(ScalaTestWithContext1.scala:7) at org.scalatest.BeforeAndAfterEach$$anonfun$1.apply$mcV$sp(BeforeAndAfterEach.scala:234)

示例代码:

  import org.apache.spark.sql.SparkSession
  import testUtils.ScalaTestWithContext1

  class SampLeTest extends ScalaTestWithContext1 {
  override protected def spark: SparkSession = ???

     test("test") {
        1 == 1 shouldBe true
     }
  }

ScalaTestWithContext1.scala

ScalaTestWithContext1.scala

  import org.apache.spark.sql.QueryTest
  import org.apache.spark.sql.test.SharedSQLContext
  import org.scalatest.{BeforeAndAfterAll, Matchers}

  abstract class ScalaTestWithContext extends QueryTest with SharedSQLContext with Matchers with BeforeAndAfterAll{}

build.sbt :

build.sbt :

name := "test"
version := "1.0"
scalaVersion := "2.11.11"

parallelExecution in Test := false

libraryDependencies ++= Seq(
  "org.scala-lang" % "scala-library" % "2.11.11" % "provided",
  "org.apache.spark" %% "spark-core" % "2.2.0",
  "org.apache.spark" %% "spark-sql" % "2.2.0",
  "org.apache.spark" %% "spark-catalyst" % "2.2.0",
  "org.apache.spark" %% "spark-core" % "2.2.0" % "test" classifier 
"tests",
  "org.apache.spark" %% "spark-sql" % "2.2.0" % "test" classifier 
"tests",
  "org.apache.spark" %% "spark-catalyst" % "2.2.0" % "test" classifier 
"tests",
  "org.scalatest" %% "scalatest" % "3.0.1" % "test"
) 

ScalaTestWithContext1 类扩展了 SharedSQLContext 和所有必需的特征.

The class ScalaTestWithContext1 extends SharedSQLContext and all required traits.

提前致谢.

推荐答案

我遇到了类似的问题.对我有用的解决方案是使用 Scalatest 的 2.2.6 版本而不是 3.x 版本.Maven 存储库也显示了正确的依赖关系在测试依赖项"部分.

I faced a similar issue. The solution that worked for me was to use version 2.2.6 of Scalatest instead of the 3.x version. The Maven repository also shows the correct dependency in the section "Test Dependencies".

这篇关于无法在 Spark-2.2.0 - Scala-2.11.8 上运行单元测试(scalatest)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！