远程Spark作业失败：找不到程序集 [英] Remote Spark Job fails: No assemblies found

问题描述

我正在使用以下conf在Vanilla Spark（Datastax）中运行Spark作业：

I am running a Spark job in a Vanilla Spark (Datastax) with this conf:

val conf: SparkConf = new SparkConf()
  .setAppName("Fuzzy")
  .setMaster("spark://127.0.0.1:7077")
  .set("spark.cores.max", "2")
  .setJars(Seq("my-jar"))

val sc: SparkContext = SparkContext.getOrCreate(conf)


val NUM_SAMPLES: Int = 500

val join = sc.parallelize(1 to NUM_SAMPLES).filter { _ =>
  val x = math.random
  val y = math.random
  x*x + y*y < 1
}.count()
println(s"Pi is roughly ${4.0 * join / NUM_SAMPLES}")

这是一个Spark示例。

This is an Spark example.

当我与local [2]一起运行时很好，但是当我使用外部Spark时

When I run with with local[2] it's fine, but when I use a external Spark it fails.

17/04/24 14:44:20信息SparkDeploySchedulerBackend：要求删除不存在的执行程序139
17/04/24 14： 44:20 INFO SparkDeployScheduler后端：授予执行器ID app-20170424144418-0012 / 140在hostPort 127.0.0.1:51227上具有2个内核，512.0 MB RAM
17/04/24 14:44:20 INFO AppClient $ ClientEndpoint：执行已更新：app-20170424144418-0012 / 140现在正在运行
17/04/24 14:44:20 INFO AppClient $ ClientEndpoint：执行程序已更新：app-20170424144418-0012 / 140现在已失败（java.lang.IllegalStateException ：在'/Users/jta/Downloads/dse-5.0.6-2/dse-5.0.6/resources/spark/lib'中找不到程序集。）
17/04/24 14:44:20 INFO SparkDeploySchedulerBackend：执行器app-20170424144418-0012 / 140已删除：java.lang.IllegalStateException：否在 /Users/jta/Downloads/dse-5.0.6-2/dse-5.0.6/resources/spark/lib中找到的程序集。
17/04/24 14:44:20信息BlockManagerMaster：删除执行程序140请求
1

17/04/24 14:44:20 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 139 17/04/24 14:44:20 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170424144418-0012/140 on hostPort 127.0.0.1:51227 with 2 cores, 512.0 MB RAM 17/04/24 14:44:20 INFO AppClient$ClientEndpoint: Executor updated: app-20170424144418-0012/140 is now RUNNING 17/04/24 14:44:20 INFO AppClient$ClientEndpoint: Executor updated: app-20170424144418-0012/140 is now FAILED (java.lang.IllegalStateException: No assemblies found in '/Users/jta/Downloads/dse-5.0.6-2/dse-5.0.6/resources/spark/lib'.) 17/04/24 14:44:20 INFO SparkDeploySchedulerBackend: Executor app-20170424144418-0012/140 removed: java.lang.IllegalStateException: No assemblies found in '/Users/jta/Downloads/dse-5.0.6-2/dse-5.0.6/resources/spark/lib'. 17/04/24 14:44:20 INFO BlockManagerMaster: Removal of executor 140 requested 1

推荐答案

<我已经知道问题出在哪里了。我运行了不同版本的Spark / Datastax，它们都使用相同的文件夹。不知何故，Spark对此感到困惑，并停止这样做并显示该错误。

I already know what the problem was. I ran different versions of Spark/Datastax and all of them used the same folders. Somehow, Spark got confused about it and stop doing that and showing that error.

一旦我从Docker运行了它，一切都会好起来。

Once I've run it from Docker, everything went fine.

这篇关于远程Spark作业失败：找不到程序集的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！