Spark安装-错误:找不到或加载主类org.apache.spark.launcher.Main [英] Spark installation - Error: Could not find or load main class org.apache.spark.launcher.Main

问题描述

在安装Spark 2.3之后并在.bashrc中设置以下环境变量(使用gitbash)

After spark installation 2.3 and setting the following env variables in .bashrc (using gitbash)

1. HADOOP_HOME

1. HADOOP_HOME

SPARK_HOME

SPARK_HOME

PYSPARK_PYTHON

PYSPARK_PYTHON

JDK_HOME

正在执行 $ SPARK_HOME/bin/spark-submit 显示以下错误.

executing $SPARK_HOME/bin/spark-submit is displaying the following error.

错误:找不到或加载主类org.apache.spark.launcher.Main

Error: Could not find or load main class org.apache.spark.launcher.Main

我在stackoverflow和其他站点中进行了一些研究检查，但无法找出问题所在.

I did some research checking in stackoverflow and other sites, but could not figure out the problem.

执行环境

1. Windows 10企业版
2. 火花版本-2.3
3. Python版本-3.6.4

能否请您提供一些提示?

Can you please provide some pointers?

推荐答案

我收到了该错误消息.可能有几个根本原因，但这是我如何调查和解决问题的方法(在Linux上):

I had that error message. It probably may have several root causes but this how I investigated and solved the problem (on linux):

• 不是启动 spark-submit ，而是尝试使用 bash -x spark-submit 来查看哪一行失败.
• 执行该过程数次(因为spark-submit调用嵌套脚本)，直到找到名为的基础过程:就我而言，它类似于:

• instead of launching spark-submit, try using bash -x spark-submit to see which line fails.
• do that process several times ( since spark-submit calls nested scripts ) until you find the underlying process called : in my case something like :

<代码>/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp'/opt/spark-2.2.0-bin-hadoop2.7/conf/:/opt/spark-2.2.0-bin-hadoop2.7/jars/*'-Xmx1g org.apache.spark.deploy.SparkSubmit --class org.apache.spark.repl.Main --name'Spark shell'spark-shell

因此，spark-submit启动一个Java进程，并且使用/opt/spark-2.2.0-bin-hadoop2.7/中的文件找不到org.apache.spark.launcher.Main类.jars/* (请参见上面的-cp选项).我在这个jars文件夹中做了一个ls，并且计算了4个文件，而不是整个火花分配(〜200个文件).在安装过程中可能是一个问题.因此，我重新安装了spark，检查了jar文件夹，它的工作原理就像一个吊饰.

So, spark-submit launches a java process and can't find the org.apache.spark.launcher.Main class using the files in /opt/spark-2.2.0-bin-hadoop2.7/jars/* (see the -cp option above). I did an ls in this jars folder and counted 4 files instead of the whole spark distrib (~200 files). It was probably a problem during the installation process. So I reinstalled spark, checked the jar folder and it worked like a charm.

因此，您应该:

• 检查 java 命令(cp选项)
• 检查您的jars文件夹(它至少包含所有spark-*.jar吗?)

希望有帮助.

这篇关于Spark安装-错误:找不到或加载主类org.apache.spark.launcher.Main的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！