如何在 Databricks 中运行 Spark-Scala 单元测试笔记本? [英] How to run a Spark-Scala unit test notebook in Databricks?
问题描述
我正在尝试使用 scalatest.funsuite 为我的 Spark-Scala 笔记本编写单元测试代码,但带有 test() 的笔记本没有在数据块中执行.你能告诉我如何运行它吗?
I am trying to write a unit test code for my Spark-Scala notebook using scalatest.funsuite but the notebook with test() is not getting executed in databricks. Could you please let me know how can I run it?
这是相同的示例测试代码.
Here is the sample test code for the same.
import org.apache.spark.sql.{Row, SparkSession}
import org.scalatest.funsuite.AnyFunSuite
import org.scalatest.Assertions._
import org.apache.spark.eventhubs._
import com.fasterxml.uuid.Generators
import org.apache.spark.sql.functions._
import org.apache.spark.sql.{DataFrame, Row, SparkSession, Column}
class ImsLoyaltyTest extends AnyFunSuite {
test ("Date Format Checker") {
val sampleIpData = Seq(
Row("india","hello","05-03-2021","14","50"),
Row("india","hello","15-06-2021","14","50"),
Row("india","hello","05/06/2021","6100","50"),
Row("india","hello","05-31-2021","14","50")
)
val sampleIpSchema = new StructType()
.add("a", StringType)
.add("b", StringType)
.add("c", StringType)
.add("d", StringType)
.add("e", StringType)
val sampleIpDF = spark.createDataFrame(spark.sparkContext.parallelize(sampleIpData), sampleIpSchema)
assert (sampleIpDF.collectAsList() == sampleIpDF.collectAsList())
}
}
推荐答案
您需要为该测试套件显式创建对象 &执行它.在 IDE 中,您依赖于特定的运行器,但它在笔记本环境中不起作用.
You need to explicitly create the object for that test suite & execute it. In IDE you're relying on specific runner, but it doesn't work in the notebook environment.
您可以使用创建对象的 .execute
函数(docs):
You can use either the .execute
function of create object (docs):
(new ImsLoyaltyTest).execute()
但最好使用 ScalaTest shell 的 .run
方法(docs) - 您可以控制颜色输出、运行多个测试等:
but it could be better to use the .run
method of ScalaTest shell (docs) - you can control color output, run multiple tests, etc.:
import org.scalatest._
nocolor.durations.stats.run(new ImsLoyaltyTest)
这篇关于如何在 Databricks 中运行 Spark-Scala 单元测试笔记本?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!