无法在Spark-2.2.0上运行单元测试(scalatest)-Scala-2.11.8 [英] Unable to run Unit tests (scalatest) on Spark-2.2.0 - Scala-2.11.8

问题描述

无法在spark-2.2.0上使用上下文运行run scalatest

Unable to run run scalatest with context on spark-2.2.0

StackTrace:

StackTrace:

异常或错误导致运行中止:org.apache.spark.sql.test.SharedSQLContext.最终(Lorg/scalatest/concurrent/PatienceConfiguration $ Timeout; Lscala/Function0; Lorg/scalatest/concurrent/AbstractPatienceConfiguration $ PatienceConfig ;)Ljava/lang/Object; java.lang.NoSuchMethodError:org.apache.spark.sql.test.SharedSQLContext.最终(Lorg/scalatest/concurrent/PatienceConfiguration $ Timeout; Lscala/Function0; Lorg/scalatest/concurrent/AbstractPatienceConfiguration $ PatienceConfig;)Ljava/lang/Object ; 在org.apache.spark.sql.test.SharedSQLContext $ class.afterEach(SharedSQLContext.scala:92) 在testUtils.ScalaTestWithContext1.afterEach(ScalaTestWithContext1.scala:7) 在org.scalatest.BeforeAndAfterEach $$ anonfun $ 1.apply $ mcV $ sp(BeforeAndAfterEach.scala:234)

An exception or error caused a run to abort: org.apache.spark.sql.test.SharedSQLContext.eventually(Lorg/scalatest/concurrent/PatienceConfiguration$Timeout;Lscala/Function0;Lorg/scalatest/concurrent/AbstractPatienceConfiguration$PatienceConfig;)Ljava/lang/Object; java.lang.NoSuchMethodError: org.apache.spark.sql.test.SharedSQLContext.eventually(Lorg/scalatest/concurrent/PatienceConfiguration$Timeout;Lscala/Function0;Lorg/scalatest/concurrent/AbstractPatienceConfiguration$PatienceConfig;)Ljava/lang/Object; at org.apache.spark.sql.test.SharedSQLContext$class.afterEach(SharedSQLContext.scala:92) at testUtils.ScalaTestWithContext1.afterEach(ScalaTestWithContext1.scala:7) at org.scalatest.BeforeAndAfterEach$$anonfun$1.apply$mcV$sp(BeforeAndAfterEach.scala:234)

示例代码:

  import org.apache.spark.sql.SparkSession
  import testUtils.ScalaTestWithContext1

  class SampLeTest extends ScalaTestWithContext1 {
  override protected def spark: SparkSession = ???

     test("test") {
        1 == 1 shouldBe true
     }
  }

ScalaTestWithContext1.scala

ScalaTestWithContext1.scala

  import org.apache.spark.sql.QueryTest
  import org.apache.spark.sql.test.SharedSQLContext
  import org.scalatest.{BeforeAndAfterAll, Matchers}

  abstract class ScalaTestWithContext extends QueryTest with SharedSQLContext with Matchers with BeforeAndAfterAll{}

build.sbt:

build.sbt :

name := "test"
version := "1.0"
scalaVersion := "2.11.11"

parallelExecution in Test := false

libraryDependencies ++= Seq(
  "org.scala-lang" % "scala-library" % "2.11.11" % "provided",
  "org.apache.spark" %% "spark-core" % "2.2.0",
  "org.apache.spark" %% "spark-sql" % "2.2.0",
  "org.apache.spark" %% "spark-catalyst" % "2.2.0",
  "org.apache.spark" %% "spark-core" % "2.2.0" % "test" classifier 
"tests",
  "org.apache.spark" %% "spark-sql" % "2.2.0" % "test" classifier 
"tests",
  "org.apache.spark" %% "spark-catalyst" % "2.2.0" % "test" classifier 
"tests",
  "org.scalatest" %% "scalatest" % "3.0.1" % "test"
) 

ScalaTestWithContext1类扩展了SharedSQLContext和所有必需的特征.

The class ScalaTestWithContext1 extends SharedSQLContext and all required traits.

预先感谢.

推荐答案

我遇到了类似的问题.对我有用的解决方案是使用Scalatest的版本2.2.6而不是3.x版本. Maven存储库还显示了正确的依赖关系在测试依赖项"部分.

I faced a similar issue. The solution that worked for me was to use version 2.2.6 of Scalatest instead of the 3.x version. The Maven repository also shows the correct dependency in the section "Test Dependencies".

这篇关于无法在Spark-2.2.0上运行单元测试(scalatest)-Scala-2.11.8的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！