如何附加资源罐进行Spark提交? [英] How to append a resource jar for spark-submit?

问题描述

我的spark应用程序依赖于adam_2.11-0.20.0.jar，每次我必须将我的应用程序与adam_2.11-0.20.0.jar打包在一起作为胖罐子提交给spark.

My spark application depends on adam_2.11-0.20.0.jar, every time I have to package my application with adam_2.11-0.20.0.jar as a fat jar to submit to spark.

例如，我的胖罐子是myApp1-adam_2.11-0.20.0.jar，

for example, my fat jar is myApp1-adam_2.11-0.20.0.jar,

可以按以下方式提交

spark-submit --class com.ano.adam.AnnoSp myApp1-adam_2.11-0.20.0.jar

它在

线程主要" java.lang.NoClassDefFoundError:

thread "main" java.lang.NoClassDefFoundError:

org/bdgenomics/adam/rdd使用--jars

org/bdgenomics/adam/rdd using --jars

spark-submit --class com.ano.adam.AnnoSp myApp1.jar --jars adam_2.11-0.20.0.jar

我的问题是如何使用2个单独的jar提交而不将它们打包在一起

My question is how to submit using 2 separate jars without package them together

spark-submit --class com.ano.adam.AnnoSp myApp1.jar adam_2.11-0.20.0.jar

推荐答案

将所有jar添加到一个文件夹中，然后执行以下操作...

Add all jars in one folder and then do like below...

我认为更好的方法是

$SPARK_HOME/bin/spark-submit \
--driver-class-path  $(echo /usr/local/share/build/libs/*.jar | tr ' ' ',') \
--jars $(echo /usr/local/share/build/libs/*.jar | tr ' ' ',') 

通过这种方法，您不会在类路径中错误地错过任何jar，因此不会出现警告.

in this approach, you wont miss any jar by mistake in the classpath hence no warning should come.

spark-submit-jars-arguments-wants-comma-list-how-to-declare-a-directory

这篇关于如何附加资源罐进行Spark提交?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！