Windows上的Hadoop构建/安装错误 [英] Hadoop on Windows Building/ Installation Error

查看：645 发布时间：2018/5/31 18:27:57 windows hadoop msbuild building

本文介绍了Windows上的Hadoop构建/安装错误的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我试图在Windows x64（8.1和Server 2012 R2）上安装 Apache Hadoop 2.7.1 ，，但我没有达到目标：

  [信息] Apache Hadoop Common ............ ................... FAILURE [37.925 s]

在过去几天里，我认为我遵循了所有教程的步骤，例如 HadoopOnWindows ，或这一个。我做了所有的先决条件，但我谷歌搜索，我的问题的相关问题如下：

将路径设置为MSBuild.exe ： C：\Windows\Microsoft.NET\Framework64\v4.0.30319;

将路径设置为 CMake

安装Microsoft SDK 7.1并使用其Microsoft SDK 7.1 CMD运行

尝试构建： hadoop-common \src\main\winutils\libwinutils.sln 和 hadoop-common \src\main\winutils\\ \\ winutils.sln 与Visual Studio 2013，但由于编译错误，构建失败。 Visual Studio 2010甚至不想打开它们。我不知道如何纠正编译错误（如果是这样的话）

其他类似问题

我所遵循的一些类似的主题是： one ， two ，三个，四个。我主要尝试了所有建议，但仍然中断。

错误

[INFO] BUILD FAILURE [INFO] ----------------------------- ------------------------------------------- [信息]总时间：01:11分钟 [INFO]完成于：2015-07-08T15：18：29 + 02：00 [INFO] Final Memory：69M / 269M [信息] -------------------------------------------------- ---------------------- [错误]无法执行目标org.codehaus.mojo：exec-maven-plugin：1.3.1：exec （c ompile-ms-winutils）项目hadoop-common：命令执行失败。进程退出时出现错误：1（退出值：1） - > [帮助1] [错误] [错误]要查看错误的完整堆栈跟踪，请使用-e开关 ch重新运行Maven。 [错误]使用-X开关重新运行Maven以启用完整的调试日志记录。 [错误] [错误]有关错误和可能解决方案的更多信息，请阅读 d以下文章： [错误] [帮助1] http：// cwiki .apache.org / confluence / display / MAVEN / MojoExecutionE xception [错误] [错误]纠正问题后，您可以使用命令 $ b恢复编译$ b [错误] mvn<目标> -rf：hadoop-common
帮助

我不知道接下来该做什么，我觉得我尝试了一切。请帮我解决这个错误，并继续在Windows上安装Hadoop。

更新1
仔细查看错误堆栈，我看到没有找到文件 Microsoft.Cpp.props 。此外，整个文件夹 Program Files（x86）/ MSBuild 缺失。因此，我安装了Visual Studio 2010，它解决了这个问题。

当然，会出现一个新问题。我能够构建 libwinutils ，但不能构建 winutils 。我收到很多 LNK2001 错误：

错误48错误LNK1120：18个未解析的外部C：\ hdfs \ haddoop-common-project\hadoop-common\src\main\winutils\x64\Release\winutils.exe winutils 错误36错误LNK2001：无法解析的外部符号BuildServiceSecurityDescriptor C：\ hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj winutils 错误32错误LNK2001：无法解析的外部符号ChownImpl C：\ hdfs\hadoop- common-project\hadoop-common\src\main\winutils\service.obj winutils 错误40错误LNK2001：无法解析的外部符号CreateEnvironmentBlock C：\ hdfs\hadoop-common-project\ hadoop-common \src\main\winutils\task.obj winutils 错误44错误LNK2001：无法解析的外部符号CreateLogonTokenForUser C：\ hdfs\hadoop-common-project\hadoop-com mon\src\main\winutils\task.obj winutils 错误41错误LNK2001：无法解析的外部符号DestroyEnvironmentBlock C：\ hdfs\hadoop-common-project\hadoop-common\src\\ \\ main \winutils\task.obj winutils 错误37错误LNK2001：无法解析的外部符号EnableImpersonatePrivileges C：\ hdfs\hadoop-common-project\hadoop-common\src\main\winutils \service.obj winutils 错误34错误LNK2001：无法解析的外部符号GetSecureJobObjectName C：\ hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj winutils 错误38错误LNK2001：无法解析的外部符号KillTask C：\ hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj winutils 错误43错误LNK2001：无法解析的外部符号LoadUserProfileForLogon C：\ hdfs\hadoop-common-project\hadoo p-common\src\main\winutils\task.obj winutils 错误35错误LNK2001：无法解析的外部符号LogDebugMessage C：\ hdfs\hadoop-common-project\hadoop-common\ src\main\winutils\service.obj winutils 错误46错误LNK2001：无法解析的外部符号LookupKerberosAuthenticationPackageId C：\ hdfs\hadoop-common-project\hadoop-common\src\main\\ \\ winutils\task.obj winutils 错误31错误LNK2001：无法解析的外部符号MIDL_user_allocate C：\ hdfs\hadoop-common-project\hadoop-common\src\main\winutils\hadoopwinutilsvc_s .obj winutils 错误30错误LNK2001：无法解析的外部符号MIDL_user_free C：\ hdfs \hadoop-common-project \hadoop-common\src\main\winutils\hadoopwinutilsvc_s.obj winutils 错误47错误LNK2001：无法解析的外部符号RegisterWithLsa C：\ hdfs \hadoop- common-project\hadoop-common\src\main\winutils\task.obj winutils 错误33错误LNK2001：无法解析的外部符号SplitStringIgnoreSpaceW C：\ hdfs\hadoop-common-project\ hadoop-common \src\main\winutils\service.obj winutils 错误42错误LNK2001：无法解析的外部符号UnloadProfileForLogon C：\ hdfs\hadoop-common-project\hadoop-common\ src\main\winutils\task.obj winutils 错误45错误LNK2001：无法解析的外部符号UnregisterWithLsa C：\ hdfs\hadoop-common-project\hadoop-common\src\main\\ \\ winutils\task.obj winutils 错误39错误LNK2001：无法解析的外部符号wsceConfigRelativePath C：\ hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service .obj winutils
更新2
@tiho解决了男人的问题以上（我非常感谢，因为我已经花了4天时间）。现在，一个新的问题出现了：

[INFO] Apache Hadoop KMS .......... ........................ FAILURE [1.531 s] [错误]无法执行目标org.apache.maven。插件：maven-antrun-plugin：1.7：项目运行（dist）hadoop-kms：出现Ant BuildException：java.net.UnknownHostException：archive.apache.org [错误]围绕Ant部分... < get dest =downloads / apache-tomcat-6.0.41.tar.gzskipexisting =trueverbose =truesrc =http://archive.apache.org/dist/tomcat/tomcat-6 /v6.0.41/bin/apache tomcat-6.0.41.tar.gz/> ... @ 5：182在C：\ haddoop-2.7.1-src\hadoop-common project\hadoop- kms\target\antrun\build-main.xml：未知主机archive.apache.org。
我发现上述问题蹩脚的解决办法。我手动下载它，添加到文件夹中：

C：\hadoop-2.7.1-src\hadoop-common-project\hadoop- kms \downloads

C：\hadoop-2.7.1-src\hadoop-hdfs-project\hadoop-hdfs-httpfs\downloads

并从档案中删除：

C：\hadoop-2.7.1-src\hadoop -common-project\hadoop-kms\target\\ \\ antrun \build-main.xml

C：\hadoop-2.7.1-src\hadoop-hdfs-project\hadoop-hdfs-httpfs\ target \antrun\build-main.xml

以下几行代码：

< mkdir dir =downloads/> < get dest =downloads / apache-tomcat-6.0.41.tar.gzskipexisting =trueverbose =truesrc =http://archive.apache.org/dist/tomcat /tomcat-6/v6.0.41/bin/apache-tomcat-6.0.41.tar.gz\"/>
请注意，我也使用代理服务器，我在 settings.xml中设置了代理服务器，并按照@tiho提供的相同建议将PATH设置为MAVEN_OPTS。这是问题吗？除了这里提供的选项外，我还向MAVEN_OPTS添加了密码和用户名。

请注意，我可以访问： http://archive.apache.org/dist/ tomcat / tomcat-6 / v6.0.41 / bin / apache-tomcat-6.0.41.tar.gz 使用我的浏览器。

但我仍然好奇如何正确地解决问题，因为这个问题仍然存在，我需要再次做同样的解决。

更新3
这就是说，本周最幸福的时刻：

[信息] ------- ----------------------------------------- [信息]建立成功 [INFO] ------------------------------------------- ----- [信息]总时间：21:05分钟 [信息]完成于：2015-07-10T11：33：17 + 02：00 [信息] Final内存：228M / 672M [信息] -------------------------------------- ----------

解决方案
我花了整整一天的时间来构建它......微软在向后兼容方面存在很多问题。如果Hadoop转移到更新的工具链中，这将有所帮助。

以下是我必须执行的关键步骤（除了文档中提到的步骤之外）。这是使用Windows SDK编译器，而不是VS 2010（它应该可能与VS 2010，但我没有尝试过）：

卸载Visual Studio 2013（我建议卸载任何Visual Studio> = 2010） - 这是为了摆脱错误 MSBUILD：配置错误MSB4146：无法评估属性表达式$（[MSBuild] :: ValueOrDefault（'$（VCTargetsPath）'，'$（MSBuildExtensionsPath32）\Microsoft.Cpp\v4.0\V120\'））

卸载Visual Studio C ++ 2010 Redistributable（x64 + x86） - 避免安装windows SDK 7.1时出错

完全卸载.NET Framework 4.5 - 避免转换为COFF时失败：文件无效或损坏 重新启动

Ins高的.NET Framework 4.0

安装Windows SDK 7.1

我添加到我的PATH（in Windows SDK shell）我的Git安装的bin文件夹（用于GNU工具），CMake的bin文件夹以及包含protoc.exe的文件夹。另外（但大多数人不应该需要它），我必须设置Maven使用我的HTTP代理，首先编辑 settings.xml ，如Maven的文档中所示，定义环境变量 MAVEN_OPTS = -Dmy.proxy.host -Dhttp.proxyPort = my.proxy.port ，否则在构建过程中的某个时刻会出现超时。 / p>

请注意，在构建过程中我注意到了很多警告：我怀疑其中一些是由于使用Java SDK 1.8而不是1.7导致的。然而，在终于看到一个 BUILD BUILD SUCCESS 之后，我不想再用1.7来试试。

I am trying to install Apache Hadoop 2.7.1 on Windows x64 (8.1 and Server 2012 R2), but I am failing at goal:
[INFO] Apache Hadoop Common ............................... FAILURE [ 37.925 s]
In the past days, I think I followed the steps of all tutorials, like HadoopOnWindows, or this one. I did all the prerequisites, but as I Googled, the relevant ones for my problem are the following:

Set path to MSBuild.exe like: C:\Windows\Microsoft.NET\Framework64\v4.0.30319; in Path

Set path to CMake

Install Microsoft SDK 7.1 and Run with its Microsoft SDK 7.1 CMD

Tried to build: hadoop-common\src\main\winutils\libwinutils.sln and hadoop-common\src\main\winutils\winutils.sln with Visual Studio 2013, but building fails due to compiling errors. Visual Studio 2010 doesn't even want to open them. I do not know how to correct the compile errors (in case this is the problem)

Other ppl, similar problem
Some of the similar topics I am following are these: one, two, three, four. I mainly tried all the suggestions, but it still breaks.

Error
[INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:11 min [INFO] Finished at: 2015-07-08T15:18:29+02:00 [INFO] Final Memory: 69M/269M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec (c ompile-ms-winutils) on project hadoop-common: Command execution failed. Process exited with an error: 1 (Exit value: 1) -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit ch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please rea d the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionE xception [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :hadoop-common
Help
I have no idea what to do next, I feel like I tried everything. Please help me solve this error and continue with the Hadoop install on Windows.

UPDATE 1 Looking on the error stack more closely, I saw that the file Microsoft.Cpp.props was not found. Moreover, the entire folder Program Files (x86)/MSBuild was missing. Thus, I installed Visual Studio 2010 and it fixed this problem.

Of course, a new problem arise. I am able to build libwinutils, but not winutils. I get many LNK2001 errors:
Error 48 error LNK1120: 18 unresolved externals C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\x64\Release\winutils.exe winutils Error 36 error LNK2001: unresolved external symbol BuildServiceSecurityDescriptor C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj winutils Error 32 error LNK2001: unresolved external symbol ChownImpl C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj winutils Error 40 error LNK2001: unresolved external symbol CreateEnvironmentBlock C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj winutils Error 44 error LNK2001: unresolved external symbol CreateLogonTokenForUser C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj winutils Error 41 error LNK2001: unresolved external symbol DestroyEnvironmentBlock C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj winutils Error 37 error LNK2001: unresolved external symbol EnableImpersonatePrivileges C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj winutils Error 34 error LNK2001: unresolved external symbol GetSecureJobObjectName C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj winutils Error 38 error LNK2001: unresolved external symbol KillTask C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj winutils Error 43 error LNK2001: unresolved external symbol LoadUserProfileForLogon C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj winutils Error 35 error LNK2001: unresolved external symbol LogDebugMessage C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj winutils Error 46 error LNK2001: unresolved external symbol LookupKerberosAuthenticationPackageId C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj winutils Error 31 error LNK2001: unresolved external symbol MIDL_user_allocate C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\hadoopwinutilsvc_s.obj winutils Error 30 error LNK2001: unresolved external symbol MIDL_user_free C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\hadoopwinutilsvc_s.obj winutils Error 47 error LNK2001: unresolved external symbol RegisterWithLsa C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj winutils Error 33 error LNK2001: unresolved external symbol SplitStringIgnoreSpaceW C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj winutils Error 42 error LNK2001: unresolved external symbol UnloadProfileForLogon C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj winutils Error 45 error LNK2001: unresolved external symbol UnregisterWithLsa C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\task.obj winutils Error 39 error LNK2001: unresolved external symbol wsceConfigRelativePath C:\hdfs\hadoop-common-project\hadoop-common\src\main\winutils\service.obj winutils
UPDATE 2 The answer from @tiho solved the problem mentioned above (and I am very thankful, as it took me 4 days already). Now, a fresh new problem, at goal:
[INFO] Apache Hadoop KMS .................................. FAILURE [ 1.531 s] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (dist) on project hadoop-kms: An Ant BuildException has occured: java.net.UnknownHostException: archive.apache.org [ERROR] around Ant part ...<get dest="downloads/apache-tomcat-6.0.41.tar.gz" skipexisting="true" verbose="true" src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.41/bin/apache tomcat-6.0.41.tar.gz"/>... @ 5:182 in C:\hadoop-2.7.1-src\hadoop-common project\hadoop-kms\target\antrun\build-main.xml: Unknown host archive.apache.org.
I found a lame work-around to the problem above. I manually downloaded it, added to the folders:

C:\hadoop-2.7.1-src\hadoop-common-project\hadoop-kms\downloads
C:\hadoop-2.7.1-src\hadoop-hdfs-project\hadoop-hdfs-httpfs\downloads
and deleted from the files:
C:\hadoop-2.7.1-src\hadoop-common-project\hadoop-kms\target\antrun\build-main.xml
C:\hadoop-2.7.1-src\hadoop-hdfs-project\hadoop-hdfs-httpfs\target\antrun\build-main.xml
the following lines of code:
<mkdir dir="downloads"/> <get dest="downloads/apache-tomcat-6.0.41.tar.gz" skipexisting="true" verbose="true" src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.41/bin/apache-tomcat-6.0.41.tar.gz"/>
Note that I am also using proxy, I have set the proxy in the settings.xml, and set PATH to MAVEN_OPTS as suggested in the same answer from @tiho. Is this the problem? besides the options suggested there, I have also added password and username to the MAVEN_OPTS.

Note that I can access: http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.41/bin/apache-tomcat-6.0.41.tar.gz using my browser.

But I am still curious how to solve the issue correctly, because this problem continues to appear, and I need to do the same work-around again.

Update 3 That being said, the happiest moment of this week:
[INFO] ------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------ [INFO] Total time: 21:05 min [INFO] Finished at: 2015-07-10T11:33:17+02:00 [INFO] Final Memory: 228M/672M [INFO] ------------------------------------------------

解决方案
Took me the whole day to manage to build it... Microsoft has obviously lots of issues with backward compatibility. It would help if Hadoop would move to a more recent toolchain.

Here are the key steps I had to perform (beyond those mentioned in the doc). This is using the Windows SDK compiler rather than VS 2010 (it is supposed to be possible with VS 2010 but I haven't tried it):

Uninstall Visual Studio 2013 (I would recommend uninstalling any Visual Studio >= 2010) -- this is to get rid of error MSBUILD : Configuration error MSB4146: Cannot evaluate the property expression "$([MSBuild]::ValueOrDefault('$(VCTargetsPath)','$(MSBuildExtensionsPath32)\Microsoft.Cpp\v4.0\V120\'))"

Uninstall Visual Studio C++ 2010 Redistributable (x64 + x86) -- to avoid Error when installing windows SDK 7.1

Uninstall fully .NET Framework 4.5 -- to avoid Failure during conversion to COFF: file invalid or corrupt

Reboot

Install .NET Framework 4.0

Install Windows SDK 7.1

I added to my PATH (in the Windows SDK shell) the bin folder of my Git installation (for GNU tools), the bin folder of CMake, and the folder containing protoc.exe. In addition (but most people should not need it), I had to setup Maven to use my HTTP proxy, first editing settings.xml as indicated in Maven's documentation, but also by defining the environment variable MAVEN_OPTS=-Dmy.proxy.host -Dhttp.proxyPort=my.proxy.port, as otherwise there was a timeout at some point during the build.

Note that I noticed a good amount of warnings during the build: I suspect some of them to be caused by using the Java SDK 1.8 instead of 1.7. However after finally seeing a BUILD SUCCESS I didn't feel like trying again with 1.7.

这篇关于Windows上的Hadoop构建/安装错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文

Windows上的Hadoop构建/安装错误 [英] Hadoop on Windows Building/ Installation Error

问题描述

相关文章

分布式计算/Hadoop最新文章

热门教程

热门工具

登录关闭

Windows上的Hadoop构建/安装错误 [英] Hadoop on Windows Building/ Installation Error

问题描述

相关文章

分布式计算/Hadoop最新文章

热门教程

热门工具

登录 关闭

登录关闭