无法从Spark Submit中的JAR文件加载主类 [英] Cannot load main class from JAR file in Spark Submit

问题描述

我正在尝试运行Spark作业.这是我的Shell脚本，位于/home/full/path/to/file/shell/my_shell_script.sh:

I am trying to run a Spark job. This is my shell script, which is located at /home/full/path/to/file/shell/my_shell_script.sh:

confLocation=../conf/my_config_file.conf &&
executors=8 &&
memory=2G &&
entry_function=my_function_in_python &&
dos2unix $confLocation &&
spark-submit \
        --master yarn-client \
        --num-executors $executors \
        --executor-memory $memory \
        --py-files /home/full/path/to/file/python/my_python_file.py $entry_function $confLocation

运行此命令时，出现错误消息:

When I run this, I get an error that says:

错误:无法从JAR文件加载主类:/home/full/path/to/file/shell/my_function_in_python

Error: Cannot load main class from JAR file: /home/full/path/to/file/shell/my_function_in_python

我的印象是它在错误的位置(python文件位于python目录中，而不是shell目录中).

My impression here is that it is looking in the wrong place (the python file is located in the python directory, not the shell directory).

推荐答案

对我有用的是简单地传入python文件而不使用--py-files命令. 看起来像这样:

What worked for me was to simply pass in the python files without the --py-files command. Looks like this:

confLocation=../conf/my_config_file.conf &&
executors=8 &&
memory=2G &&
entry_function=my_function_in_python &&
dos2unix $confLocation &&
spark-submit \
        --master yarn-client \
        --num-executors $executors \
        --executor-memory $memory \
        /home/full/path/to/file/python/my_python_file.py $entry_function $confLocation

这篇关于无法从Spark Submit中的JAR文件加载主类的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！