为什么 ./bin/spark-shell 给 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform? [英] Why does ./bin/spark-shell give WARN NativeCodeLoader: Unable to load native-hadoop library for your platform?
问题描述
在 Mac OS X 上，我使用以下命令从源代码编译 Spark:
On Mac OS X, I compiled Spark from the sources using the following command:
jacek:~/oss/spark
$ SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true SPARK_HIVE=true SPARK_GANGLIA_LGPL=true xsbt
...
[info] Set current project to root (in build file:/Users/jacek/oss/spark/)
> ; clean ; assembly
...
[info] Packaging /Users/jacek/oss/spark/examples/target/scala-2.10/spark-examples-1.0.0-SNAPSHOT-hadoop2.4.0.jar ...
[info] Done packaging.
[info] Done packaging.
[success] Total time: 1964 s, completed May 9, 2014 5:07:45 AM
当我启动 ./bin/spark-shell 时，我注意到以下警告消息:
When I started ./bin/spark-shell I noticed the following WARN message:
警告 NativeCodeLoader:无法为您的平台...在适用的情况下使用内置 Java 类
WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
可能是什么问题?
jacek:~/oss/spark
$ ./bin/spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on classpath
14/05/09 21:11:17 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/05/09 21:11:17 INFO SecurityManager: Changing view acls to: jacek
14/05/09 21:11:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jacek)
14/05/09 21:11:17 INFO HttpServer: Starting HTTP Server
Welcome to
____ __
/ __/__ ___ _____/ /__
_ / _ / _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_ version 1.0.0-SNAPSHOT
/_/
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0)
Type in expressions to have them evaluated.
Type :help for more information.
...
14/05/09 21:11:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
...
推荐答案
支持平台 内容如下:
本机 hadoop 库仅在 *nix 平台上受支持.这库不适用于 Cygwin 或 Mac OS X 平台.
The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.
原生 hadoop 库主要用于 GNU/Linus 平台和已经在这些发行版上进行了测试:
The native hadoop library is mainly used on the GNU/Linus platform and has been tested on these distributions:
- RHEL4/Fedora
- Ubuntu
- Gentoo
在上述所有发行版中，32/64 位原生 hadoop 库将与相应的 32/64 位 jvm 配合使用.
On all the above distributions a 32/64 bit native hadoop library will work with a respective 32/64 bit jvm.
似乎在 Mac OS X 上应该忽略 WARN 消息，因为该平台不仅仅存在本机库.
It appears that the WARN message should be disregarded on Mac OS X as the native library doesn't simply exist for the platform.
这篇关于为什么 ./bin/spark-shell 给 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
