java.lang.UnsatisfiedLinkError:org.apache.hadoop.io.nativeio.NativeIO $ Windows.createDirectoryWithMode0 [英] java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0

问题描述

我无法解决此异常，我已经阅读了hadoop docu和我可以找到的所有相关的stackoverflow问题.

我的fileSystem.mkdirs(***)抛出:

Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:465)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:518)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:496)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:316)
...

我在我的应用程序中包含以下依赖项(通过maven pom.xml)，都在2.6.0-cdh5.13.0版本中:hadoop-common，hadoop-hdfs，hadoop-client，hadoop-minicluster

我的文件系统变量是有效的(hadoop通用)文件系统(org.apache.hadoop.fs.FileSystem).

我从 https://github下载了hadoop文件.com/steveloughran/winutils/tree/master/hadoop-2.6.0/bin . 我将版本2.6.0中的winutils.exe和所有其他文件存储到C:\ Temp \ hadoop \ bin下的本地文件系统中. 我在路径变量HADOOP_HOME中添加了C:\ Temp \ hadoop(是的，不是bin目录的路径).

未使用后备广告(使用内建的Java类")，我得到了:

145 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader  - Trying to load the custom-built native-hadoop library...
147 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader  - Loaded the native-hadoop library

(请参见 https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/NativeLibraries.html )

我了解到，此异常可能是由hadoop版本不匹配引起的，但是我检查了导入的hadoop是否与我本地存储的hadoop匹配，版本明智.

我正在Windows 10 x64系统和IntelliJ中工作.

每个人都有一个主意，我可以检查甚至是我做错了什么?

更新: 我使用以下VM选项运行我的主服务器

-Dhadoop.home.dir=C:/Temp/hadoop
-Djava.library.path=C:/Temp/hadoop/bin

不指定lib路径，我得到:

org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

解决方案

对我来说，设置VM参数-Djava.library.path = C:\ devTools \ winutils-master \ hadoop-3.0.0 解决了这个问题.

I cannot solve this exception, I've read the hadoop docu and all related stackoverflow questions that I could find.

My fileSystem.mkdirs(***) throws:

Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Ljava/lang/String;I)V
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode0(Native Method)
    at org.apache.hadoop.io.nativeio.NativeIO$Windows.createDirectoryWithMode(NativeIO.java:524)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:465)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:518)
    at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:496)
    at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:316)
...

I am including the following dependencies in my app (via maven pom.xml), all in version 2.6.0-cdh5.13.0: hadoop-common, hadoop-hdfs, hadoop-client, hadoop-minicluster

My filesystem variable is a valid (hadoop-common) FileSystem (org.apache.hadoop.fs.FileSystem).

I downloaded the hadoop files from https://github.com/steveloughran/winutils/tree/master/hadoop-2.6.0/bin. I stored the winutils.exe and all the other files from in version 2.6.0 to my local file system under C:\Temp\hadoop\bin. I added the path variable HADOOP_HOME with C:\Temp\hadoop (yes, not the path to the bin directory).

The fallback is not used ("using builtin-java classes"), I am getting:

145 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader  - Trying to load the custom-built native-hadoop library...
147 [main] DEBUG org.apache.hadoop.util.NativeCodeLoader  - Loaded the native-hadoop library

(See https://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/NativeLibraries.html)

I understood, that this exception can be caused by a hadoop version mismatch, but I checked that the imported hadoop matches the hadoop I stored locally, version wise.

I am working on a Windows 10 x64 system and in IntelliJ.

Anybody has an idea, what I could check or even, what I am doing wrong?

UPDATE: I run my main with the following VM options

-Dhadoop.home.dir=C:/Temp/hadoop
-Djava.library.path=C:/Temp/hadoop/bin

Without specifying the lib path, I get:

org.apache.hadoop.util.NativeCodeLoader  - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

解决方案

To me setting VM Argument -Djava.library.path=C:\devTools\winutils-master\hadoop-3.0.0 resolved the issue.

这篇关于java.lang.UnsatisfiedLinkError:org.apache.hadoop.io.nativeio.NativeIO $ Windows.createDirectoryWithMode0的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！