如何从IntelliJ（或其他IDE）运行Spark测试 [英] How to run a Spark test from IntelliJ (or other IDE)

问题描述

我试图为一些Spark代码创建一个测试。获取SparkSession对象时，以下代码失败。注意：从cli运行测试运行正常：gradle my_module：build

  @Test 
 def myTest（） ：Unit = {
 val spark = SparkSession.builder（）。master（local [2]）。getOrCreate（）
 ... 
} 
  
 
 $ b 
错误： 
 
 
  java.lang .IllegalArgumentException：无法获取Kerberos领域
 ... 
引起：java.lang.reflect.InvocationTargetException 
 ... 
引起：KrbException：找不到默认领域
  
我的设置：IntelliJ + Gradle + Mac OS  
 
 
问题：
 
 
 
 
如何在IntelliJ中运行Spark测试？
 
 
为什么Spark在运行'local'时寻找Kerberos 
 
 
 
 
解决方案
通过您的代码你需要从JUnit运行Spark，而不是专门从IntelliJ运行，你可以尝试像 https://github.com/sleberknight/sparkjava-testing  
 
I am trying to create a Test for some Spark code. The following code fails when getting a SparkSession object. NOTE: The test runs fine when running from the cli: gradle my_module:build
@Test
def myTest(): Unit = {
  val spark = SparkSession.builder().master("local[2]").getOrCreate()
  ...
}
Error:
java.lang.IllegalArgumentException: Can't get Kerberos realm
...
Caused by: java.lang.reflect.InvocationTargetException
...
Caused by: KrbException: Cannot locate default realm
My set-up: IntelliJ + Gradle + Mac OS

Questions:


How do I run a Spark Test from within IntelliJ?
Why is Spark looking for Kerberos at all when running 'local'

解决方案
By your code you need to run Spark from JUnit, not specifically from IntelliJ, you can try something like https://github.com/sleberknight/sparkjava-testing

                        
这篇关于如何从IntelliJ（或其他IDE）运行Spark测试的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！