星火工人无法找到EC2集群上JAR [英] Spark workers unable to find JAR on EC2 cluster

问题描述

我使用的火花EC2运行一些星火code。当我设置大师
本地，那么它​​运行良好。但是，当我设置师傅$ MASTER，
工人们立即失败，与java.lang.NoClassDefFoundError的对
类。工人们连接到主，并在用户界面显示出来，并尝试运行任务;但马上，尽快提高该例外，因为它加载它的第一个相关性类（这是在组装罐）。

I'm using spark-ec2 to run some Spark code. When I set master to "local", then it runs fine. However, when I set master to $MASTER, the workers immediately fail, with java.lang.NoClassDefFoundError for the classes. The workers connect to the master, and show up in the UI, and try to run the task; but immediately raise that exception as soon as it loads its first dependency class (which is in the assembly jar).

我用SBT-组装做出罐子类，使用确认
罐子TVF的类在那里，并设置SparkConf分配
类。星火的Web UI确实显示了组装罐子是
添加到类路径：
<一href=\"http://172.x.x.x47441/jars/myjar-assembly-1.0.jar\">http://172.x.x.x47441/jars/myjar-assembly-1.0.jar

I've used sbt-assembly to make a jar with the classes, confirmed using jar tvf that the classes are there, and set SparkConf to distribute the classes. The Spark Web UI indeed shows the assembly jar to be added to the classpath: http://172.x.x.x47441/jars/myjar-assembly-1.0.jar

看来，尽管myjar这一组件包含
类，并且被添加到集群，这不是在到达
工人。我该如何解决这个问题？ （我需要手动复制jar文件？
如果是这样，向其中迪尔？我认为SparkConf的加点
罐子是自动执行此操作）

It seems that, despite the fact that myjar-assembly contains the class, and is being added to the cluster, it's not reaching the workers. How do I fix this? (Do I need to manually copy the jar file? If so, to which dir? I thought that the point of the SparkConf add jars was to do this automatically)

我在调试的尝试已经证明：

My attempts at debugging have shown:

1. 组装罐正在被复制到/根/火花/工作/ APP-XXXXXX / 1 /
   的（通过ssh来测定工人和搜索JAR）的


2. 然而，这条道路不会出现在工人的类路径
   的（原木，这表明java命令，但缺乏该文件而定）的

1. The assembly jar is being copied to /root/spark/work/app-xxxxxx/1/ (Determined by ssh to worker and searching for jar)
2. However, that path doesn't appear on the worker's classpath (Determined from logs, which show java -cp but lack that file)

所以，好像我要告诉火花的路径添加到装配
罐子工人的类路径中。我怎么做？还是有另外一个罪魁祸首？ （我花了几个小时试图调试这一点，但没有用！）

So, it seems like I need to tell Spark to add the path to the assembly jar to the worker's classpath. How do I do that? Or is there another culprit? (I've spent hours trying to debug this but to no avail!)

推荐答案

请注意：EC2具体的答案，不是一般的星火答案。只是想一个答案圆了一个问题问一年前，一个有相同的症状，但往往不同的原因和旅行了很多人。

NOTE: EC2 specific answer, not a general Spark answer. Just trying to round out an answer to a question asked a year ago, one that has the same symptom but often different causes and trips up a lot of people.

如果我理解正确的问题，你问，我需要手动复制jar文件？如果是这样，哪个目录？你说，并设置SparkConf分发班，但如果这是通过spark-env.sh或火花defaults.conf做你是不是清楚了吗？所以做一些假设，主要一个是你在集群模式下运行，这意味着你的驱动器上的一个工人跑，你不知道这在中前...然后... ...

If I am understanding the question correctly, you are asking, "Do I need to manually copy the jar file? If so, to which dir?" You say, "and set SparkConf to distribute the classes" but you are not clear if this is done via spark-env.sh or spark-defaults.conf? So making some assumptions, the main one being your are running in cluster mode, meaning your driver runs on one of the workers and you don't know which one in advance... then...

答案是肯定的，在类路径中指定的目录。在EC2中唯一持久性数据存储/根/持久HDFS，但我不知道这是一个好主意。

The answer is yes, to the dir named in the classpath. In EC2 the only persistent data storage is /root/persistent-hdfs, but I don't know if that's a good idea.

在EC2上的星火文档我看到这一行：

To deploy code or data within your cluster, you can log in and use
the provided script ~/spark-ec2/copy-dir, which, given a directory 
path, RSYNCs it to the same location on all the slaves.

SPARK_CLASSPATH

我不会用SPARK_CLASSPATH因为它是pcated星火1.0这么一个好主意，就是用它在$ SPARK_HOME更换德$ P $ / conf资料/火花defaults.conf：

I wouldn't use SPARK_CLASSPATH because it's deprecated as of Spark 1.0 so a good idea is to use its replacement in $SPARK_HOME/conf/spark-defaults.conf:

spark.executor.extraClassPath /path/to/jar/on/worker

这应该是工作的选项。如果你需要做的飞行，而不是在conf文件中，建议./spark-submit与--driver类路径来增强驾驶员的classpath（的星火文档相对=nofollow>，看看答案结束对另一个源）。

This should be the option that works. If you need to do this on the fly, not in a conf file, the recommendation is "./spark-submit with --driver-class-path to augment the driver classpath" (from Spark docs about spark.executor.extraClassPath and see end of answer for another source on that).

但你不使用火花提交......我不知道如何在EC2的作品，在看剧本我也没弄清楚哪里EC2使你可以在命令行中提供这些参数。你提到你已经设置您SparkConf对象与这样坚持下去是否适合你做这个。

BUT ... you are not using spark-submit ... I don't know how that works in EC2, looking at the script I didn't figure out where EC2 let's you supply these parameters on a command line. You mention you already do this in setting up your SparkConf object so stick with that if that works for you.

我星火年看到这是一个非常古老的问题，所以我不知道你是如何解决的呢？我希望这可以帮助别人，我学到了很多研究EC2的细节。

I see in Spark-years this is a very old question so I wonder how you resolved it? I hope this helps someone, I learned a lot researching the specifics of EC2.

我必须承认，作为这个限制，它混淆了我在的星火文档，对于spark.executor.extraClassPath 的它说：

I must admit, as a limitation on this, it confuses me in the Spark docs that for spark.executor.extraClassPath it says:

用户通常不应该需要设置这个选项

Users typically should not need to set this option

我想他们的意思是大多数人会得到classpath中出通过驱动程序配置选项。我知道大多数的文档的火花提交使它想脚本处理移动你的code群周围，但我认为这只是在独立客户端模式，我以为你不使用，我认为EC2必须在独立群集模式。

I assume they mean most people will get the classpath out through a driver config option. I know most of the docs for spark-submit make it should like the script handles moving your code around the cluster but I think that's only in "standalone client mode" which I assume you are not using, I assume EC2 must be in "standalone cluster mode."

更多/背景ON SPARK_CLASSPATH德precation：

MORE / BACKGROUND ON SPARK_CLASSPATH deprecation:

更多的背景下，使我认为SPARK_CLASSPATH <一个href=\"https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3C01a901d0547c$a23ba480$e6b2ed80$@innowireless.com%3E\"相对=nofollow>是德precated是这样的归档线程。和<一个href=\"https://mail-archives.apache.org/mod_mbox/spark-user/201502.mbox/[e-mail]%3E\"相对=nofollow>这一项，穿越其他线程并的这一项关于使用SPARK_CLASSPATH 时显示警告信息：

More background that leads me to think SPARK_CLASSPATH is deprecated is this archived thread. and this one, crossing the other thread and this one about a WARN message when using SPARK_CLASSPATH:

14/07/09 13:37:36 WARN spark.SparkConf:
SPARK_CLASSPATH was detected (set to 'path-to-proprietary-hadoop-lib/*:
/path-to-proprietary-hadoop-lib/lib/*').
This is deprecated in Spark 1.0+.

Please instead use:
 - ./spark-submit with --driver-class-path to augment the driver classpath
 - spark.executor.extraClassPath to augment the executor classpath

这篇关于星火工人无法找到EC2集群上JAR的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！