SBT测试不适用于火花测试 [英] SBT test does not work for spark test

问题描述

我有一个简单的spark函数来测试DF窗口:

I have a simple spark function to test DF windowing:

    import org.apache.spark.sql.{DataFrame, SparkSession}

    object ScratchPad {

      def main(args: Array[String]): Unit = {
        val spark = SparkSession.builder().master("local[*]").getOrCreate()
        spark.sparkContext.setLogLevel("ERROR")
        get_data_frame(spark).show()
      }

      def get_data_frame(spark: SparkSession): DataFrame = {
        import spark.sqlContext.implicits._
        val hr = spark.sparkContext.parallelize(List(
          ("Steinbeck", "Sales", 100),
          ("Woolf", "IT", 99),
          ("Wodehouse", "Sales", 250),
          ("Hemingway", "IT", 349)
        )
        ).toDF("emp", "dept", "sal")

        import org.apache.spark.sql.expressions.Window
        import org.apache.spark.sql.functions._

        val windowspec = Window.partitionBy($"dept").orderBy($"sal".desc)


        hr.withColumn("rank", row_number().over(windowspec))

      }
    }

我这样写了一个测试:

    import com.holdenkarau.spark.testing.DataFrameSuiteBase
    import org.apache.spark.sql.Row
    import org.apache.spark.sql.types._
    import org.scalatest.FunSuite

    class TestDF extends FunSuite with DataFrameSuiteBase  {

      test ("DFs equal") {
        val expected=sc.parallelize(List(
          Row("Wodehouse","Sales",250,1),
          Row("Steinbeck","Sales",100,2),
          Row("Hemingway","IT",349,1),
          Row("Woolf","IT",99,2)
        ))

        val schema=StructType(
          List(
          StructField("emp",StringType,true),
          StructField("dept",StringType,true),
          StructField("sal",IntegerType,false),
          StructField("rank",IntegerType,true)
          )
        )

        val e2=sqlContext.createDataFrame(expected,schema)
        val actual=ScratchPad.get_data_frame(sqlContext.sparkSession)
        assertDataFrameEquals(e2,actual)
      }

}

当我右键单击intellij中的类并单击运行"时，效果很好. 当我使用"sbt测试"运行相同的测试时，它失败，并显示以下内容:

Works fine when I right click on the class in intellij and click "run". When I run the same test with "sbt test", it fails with the following:

    java.security.AccessControlException: access denied 
    org.apache.derby.security.SystemPermission( "engine", 
    "usederbyinternals" )
        at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
        at java.security.AccessController.checkPermission(AccessController.java:884)
        at org.apache.derby.iapi.security.SecurityUtil.checkDerbyInternalsPrivilege(Unknown Source)
        ...

这是我的SBT脚本，没有任何花哨的内容可以放入蜂巢依赖性，否则测试将无法编译:

And here is my SBT script, nothing fancy-had to put in hive dependency, otherwise the test would not compile:

    name := "WindowingTest"

    version := "0.1"

    scalaVersion := "2.11.5"


    libraryDependencies += "org.apache.spark" %% "spark-core" % "2.2.1"
    libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.2.1"
    libraryDependencies += "org.apache.spark" %% "spark-hive" % "2.2.1"
    libraryDependencies += "com.holdenkarau" %% "spark-testing-base" % "2.2.0_0.8.0" % "test"

Google搜索将我指向derby-6648( https://db .apache.org/derby/releases/release-10.12.1.1.cgi )

Google search points me to derby-6648 (https://db.apache.org/derby/releases/release-10.12.1.1.cgi)

其中表示: 需要进行申请变更 在SecurityManager下运行Derby的用户必须编辑策略文件，并将以下附加权限授予derby.jar，derbynet.jar和derbyoptionaltools.jar:

which says: Application Changes Required Users who run Derby under a SecurityManager must edit the policy file and grant the following additional permission to derby.jar, derbynet.jar, and derbyoptionaltools.jar:

permission org.apache.derby.security.SystemPermission"engine"，"usederbyinternals";

permission org.apache.derby.security.SystemPermission "engine", "usederbyinternals";

由于我没有明确安装derby(可能由spark内部使用)，我该怎么做?

Since I did not explicitly install derby (probably used by spark internally), how do I do this?

推荐答案

快速又肮脏的黑客攻击解决了问题

Folowing quick and dirty hack solves the problem

System.setSecurityManager(null)

无论如何，因为它仅与自动化测试有关，也许毕竟没有那么大问题；)

Anyway as it's related to automated tests only maybe it's not that problematic after all ;)

这篇关于SBT测试不适用于火花测试的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！