为什么./bin/spark-shell提供WARN NativeCodeLoader：无法为您的平台加载native-hadoop库？ [英] Why does ./bin/spark-shell give WARN NativeCodeLoader: Unable to load native-hadoop library for your platform?

问题描述

在Mac OS X上，我使用以下命令从源代码编译了Spark：

  jacek：〜/ oss / spark 
 $ SPARK_HADOOP_VERSION = 2.4.0 SPARK_YARN = true SPARK_HIVE = true SPARK_GANGLIA_LGPL = true xsbt 
 ... 
 
 [info]将当前项目设置为root（在build file：/ Users / jacek / oss / spark /）
> ;干净汇编
 ... 
 [info]打包/Users/jacek/oss/spark/examples/target/scala-2.10/spark-examples-1.0.0-SNAPSHOT-hadoop2.4.0.jar .. 。
 [info]完成打包。 
 [info]完成打包。 
 [成功]总时间：1964 s，2014年5月9日上午5:07:45完成
  

$ b $ b

WARN NativeCodeLoader：无法为您的
平台加载native-hadoop库...在适用的情况下使用builtin-java类

可能是什么问题？

  jacek：〜/ oss / spark 
 $ ./bin / spark-shell 
 Spark组件使用Hive构建，包括类路径上的Datanucleus jar 
 14/05/09 21:11:17 INFO SecurityManager：使用Spark的默认log4j配置文件：org / apache / spark / log4j-defaults.properties 
 14/05/09 21:11:17 INFO SecurityManager：将视图acls更改为：jacek 
 14/05/09 21:11:17 INFO SecurityManager：SecurityManager：身份验证已禁用;用户禁用;具有查看权限的用户：Set（jacek）
 14/05/09 21:11:17 INFO HttpServer：启动HTTP服务器
欢迎使用
 ____ __ 
 / __ / __ ___ _____ / / __ 
 _ \ \\ / \\ \\ \\ / _`/ __ /'_ / 
 / ___ / .__ / \ _，_ / _ / / _ / \_ \ version 1.0.0-SNAPSHOT 
 / _ / 
 
使用Scala版本2.10.4（Java HotSpot™64位服务器虚拟机，Java 1.8.0）
键入表达式以评估它们。 
输入：help获取更多信息。 
 ... 
 14/05/09 21:11:49 WARN NativeCodeLoader：无法为您的平台加载native-hadoop库...在适用的情况下使用builtin-java类
 .. 。
  

解决方案

支持的平台内容如下：

本地hadoop库仅在* nix平台上受支持。
库不适用于Cygwin或Mac OS X平台。

原生hadoop库主要用于GNU / Linus平台，
已经在这些发行版上进行了测试：

• RHEL4 / Fedora


• Ubuntu


• Gentoo

在以上所有发行版中，32/64位本机hadoop库将分别与32个/ 64位jvm。

看起来WARN消息在Mac OS X上应该被忽略，因为本地库不仅仅存在为平台。

On Mac OS X, I compiled Spark from the sources using the following command:

jacek:~/oss/spark
$ SPARK_HADOOP_VERSION=2.4.0 SPARK_YARN=true SPARK_HIVE=true SPARK_GANGLIA_LGPL=true xsbt
...

[info] Set current project to root (in build file:/Users/jacek/oss/spark/)
> ; clean ; assembly
...
[info] Packaging /Users/jacek/oss/spark/examples/target/scala-2.10/spark-examples-1.0.0-SNAPSHOT-hadoop2.4.0.jar ...
[info] Done packaging.
[info] Done packaging.
[success] Total time: 1964 s, completed May 9, 2014 5:07:45 AM

When I started ./bin/spark-shell I noticed the following WARN message:

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

What might be the issue?

jacek:~/oss/spark
$ ./bin/spark-shell
Spark assembly has been built with Hive, including Datanucleus jars on classpath
14/05/09 21:11:17 INFO SecurityManager: Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
14/05/09 21:11:17 INFO SecurityManager: Changing view acls to: jacek
14/05/09 21:11:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jacek)
14/05/09 21:11:17 INFO HttpServer: Starting HTTP Server
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.0.0-SNAPSHOT
      /_/

Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0)
Type in expressions to have them evaluated.
Type :help for more information.
...
14/05/09 21:11:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
...

解决方案

Supported Platforms of the Native Libraries Guide documentation in Apache Hadoop reads:

The native hadoop library is supported on *nix platforms only. The library does not to work with Cygwin or the Mac OS X platform.

The native hadoop library is mainly used on the GNU/Linus platform and has been tested on these distributions:

• RHEL4/Fedora
• Ubuntu
• Gentoo

On all the above distributions a 32/64 bit native hadoop library will work with a respective 32/64 bit jvm.

It appears that the WARN message should be disregarded on Mac OS X as the native library doesn't simply exist for the platform.

这篇关于为什么./bin/spark-shell提供WARN NativeCodeLoader：无法为您的平台加载native-hadoop库？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！