没有Hadoop的Spark：无法启动 [英] Spark without Hadoop: Failed to Launch

问题描述

我在Ubuntu 16.04上运行Spark 2.1.0，Hive 2.1.1和Hadoop 2.7.3。

我从github下载Spark项目并构建没有hadoop版本：

./ dev / make-distribution.sh --namehadoop2-without-hive--tgz
-Pyarn，hadoop-provided，hadoop-2.7，parquet-provided

当我运行<$ c $

  Spark命令：/ usr / lib / jvm / java-8 -openjdk-amd64 / jre / bin / java -cp / home / server / spark / conf /：/ home / server / spark / jars / *：/ home / server / Hadoop的在/ etc / Hadoop的/：/首页/服务器/ Hadoop的/股/的Hadoop / common / lib目录/：/首页/服务器/ Hadoop的/股/的Hadoop /普通/：/首页/服务器/ Hadoop的/股/的Hadoop / MapReduce的/ ：/ home / server / hadoop / share / hadoop / mapreduce / lib /：/ home / server / hadoop / share / hadoop / yarn /：/ home / server / hadoop / share / hadoop / yarn / lib / -Xmx1g org。 apache.spark.deploy.master.Master --host ThinkPad-W550s-Lab --port 7077 --webui-port 8080 
 ===== =================================== 
错误：发生了JNI错误，请检查您的安装并重试
线程main中的异常java.lang.NoClassDefFoundError：org / slf4j / Logger $ b $ at java.lang.Class.getDeclaredMethods0（Native Method）$ b $ java.util。 Class.privateGetDeclaredMethods（Class.java:2701）
 at java.lang.Class.privateGetMethodRecursive（Class.java:3048）$ b $ java.util.Class.getMethod0（Class.java:3018）
 at java.lang.Class.getMethod（Class.java:1784）
 at sun.launcher.LauncherHelper.validateMainClass（LauncherHelper.java:544）
 at sun.launcher.LauncherHelper.checkAndLoadMain（LauncherHelper .java：526）
导致：java.lang.ClassNotFoundException：org.slf4j.Logger $ b $ java.net.URLClassLoader.findClass（URLClassLoader.java:381）
 at java.lang .ClassLoader.loadClass（ClassLoader.java:424）
 at sun.misc.Launcher $ AppClassLoader.loadClass（Launcher.java:331）
 at java.lang.Class Loader.loadClass（ClassLoader.java:357）
 ... 7 more 
  

我编辑SPARK_DIST_CLASSPATH根据发布的 hadoop 2中的hadoop jar文件在哪里？

  export SPARK_DIST_CLASSPATH =〜/ hadoop / share / hadoop / common / lib：〜/ hadoop / share / hadoop / common ：〜/ hadoop / share / hadoop / yarn / lib 
 / hadoop / share / hadoop / share / hadoop / code> 

但是我仍然遇到同样的错误。
我可以看到slf4j jar文件位于〜/ hadoop / share / hadoop / common / lib 。

Hadoop free构建需要修改SPARK_DIST_CLASSPATH以包含Hadoop的包jar。最方便的地方是在conf / spark-env.sh中添加一个条目：

export SPARK_DIST_CLASSPATH = $（/ path / to / hadoop / bin / hadoop classpath）

请检查此 https://spark.apache.org/docs/latest/ hadoop-provided.html

I'm running Spark 2.1.0, Hive 2.1.1 and Hadoop 2.7.3 on Ubuntu 16.04.

I download the Spark project from github and build the "without hadoop" version:

./dev/make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.7,parquet-provided"

When I run ./sbin/start-master.sh, I get the following exception:

 Spark Command: /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -cp /home/server/spark/conf/:/home/server/spark/jars/*:/home/server/hadoop/etc/hadoop/:/home/server/hadoop/share/hadoop/common/lib/:/home/server/hadoop/share/hadoop/common/:/home/server/hadoop/share/hadoop/mapreduce/:/home/server/hadoop/share/hadoop/mapreduce/lib/:/home/server/hadoop/share/hadoop/yarn/:/home/server/hadoop/share/hadoop/yarn/lib/ -Xmx1g org.apache.spark.deploy.master.Master --host ThinkPad-W550s-Lab --port 7077 --webui-port 8080
 ========================================
 Error: A JNI error has occurred, please check your installation and try again
 Exception in thread "main" java.lang.NoClassDefFoundError: org/slf4j/Logger
     at java.lang.Class.getDeclaredMethods0(Native Method)
     at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
     at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
     at java.lang.Class.getMethod0(Class.java:3018)
     at java.lang.Class.getMethod(Class.java:1784)
     at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
     at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
 Caused by: java.lang.ClassNotFoundException: org.slf4j.Logger
     at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
     at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
     ... 7 more

I edit SPARK_DIST_CLASSPATH according to the post Where are hadoop jar files in hadoop 2?

export SPARK_DIST_CLASSPATH=~/hadoop/share/hadoop/common/lib:~/hadoop/share/hadoop/common:~/hadoop/share/hadoop/mapreduce:~/hadoop/share/hadoop/mapreduce/lib:~/hadoop/share/hadoop/yarn:~/hadoop/share/hadoop/yarn/lib

But I'm still getting the same error. I can see the slf4j jar file is under ~/hadoop/share/hadoop/common/lib.

How could I fix this error?

Thank you!

解决方案

"Hadoop free" builds need to modify SPARK_DIST_CLASSPATH to include Hadoop’s package jars. The most convenient place to do this is by adding an entry in conf/spark-env.sh :
export SPARK_DIST_CLASSPATH=$(/path/to/hadoop/bin/hadoop classpath)
check this https://spark.apache.org/docs/latest/hadoop-provided.html

这篇关于没有Hadoop的Spark：无法启动的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！