Wordcount C ++ Hadoop管道不起作用 [英] Wordcount C++ Hadoop pipes does not work
问题描述
我试图在C ++中运行wordcount的例子，就像这个链接描述的那样：
使用C ++运行WordCount程序。编译工作正常，但是当我试图运行我的程序时，出现了一个错误：
blockquote
bin / hadoop pipes -conf ../dev /word.xml - 输入testtile.txt - 输出wordcount-out
11/06/06 14:23:40 WARR mapred.JobClient：没有作业jar文件集。用户类可能找不到。请参阅JobConf（Class）或JobConf＃setJar（String）。
11/06/06 14:23:40信息mapred.FileInputFormat：要处理的总输入路径：1
11 / 06/06 14:23:40信息mapred.JobClient：正在运行的作业：job_201106061207_0007
11/06/06 14:23:41信息mapred.JobClient：map 0％reduce 0％
11/06/06 14:23:53信息mapred.JobClient：任务ID：attempt_201106061207_0007_m_000000_0，状态：FAILED
java.io.IOException
at org.apache。 hadoop.mapred.pipes.OutputHandler.waitForAuthentication（OutputHandler.java:188）
at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication（Application.java:194）
at org.apache。 hadoop.mapred.pipes.Application。（Application.java:149）
at org.apache.hadoop.mapred.pipes.PipesMapRunner.run（PipesMapRunner.java:68）
at org.apache.hadoop .mapred.MapTask.runOldMapper（MapTask.java:435）
at org.apache.hadoop.mapred.MapTask.run（MapTask.java:371）
at org.apache.hadoop.mapred.Child $ 4.run（Child.java:259）$ b $ at java.security.AccessController.doPrivileged（Native Method）
at javax.security.auth.Subject.doAs （Subject.java:416）
at org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1059）
at org.apache.hadoop.mapred.Child.main（Child.java ：253）
attempt_201106061207_0007_m_000000_0：服务器未通过身份验证。退出
我在两个节点上运行Fedora上的Hadoop，并遵循指示从该链接进行配置：在多节点群集上运行Hadoop 。我用这个命令尝试了Hadoop的wordcount示例：
lockquote
bin / hadoop jar hadoop-examples-0.20.203.0.jar wordcount testtile.txt wordcount-out
并且该命令正常工作。这就是为什么我不明白为什么我的程序不起作用。所以我希望有人知道我做错了什么，或者有人已经解决了这个错误。
不知道我是否必须以这种方式回答我的问题，或者编辑我的问题。无论如何，我找到了解决方案，我只是想告诉大家谁会得到相同的错误。经过几天的研究和尝试，我明白，Fedora和C ++对于64位Hadoop来说不是一个好的选择。我试图编译Hadoop wordcount C ++，像在wiki中解释的那样。但是，蚂蚁给了我一些关于：libssl和stdint的错误。
首先，如果你使用Fedora，你必须在 -lcrypto 强> LIBS 变量放在 .configure 中。这是因为在链接到libssl时，现在必须在这些平台上明确声明对libcrypto的依赖（请参阅错误第二个问题：ant会产生很多关于C ++文件的错误：要解决这个问题，你只需要添加一个include： stdint.h 放在文件的顶部。然后构建成功。然后我尝试在我的Hadoop集群上运行wordcount示例，并且它可以正常工作，而我却没有。我希望这个问题来自我刚刚纠正的库，我是对的：我试图从hadoop安装目录运行Hadoop示例库，但它没有工作，并且出现相同的错误。
这可以通过ant重新编译Hadoop所需的C ++库（并进行修正）并使用它来解释，而不是库在Hadoop安装目录中提供。
I am trying to run the example of wordcount in C++ like this link describes the way to do : Running the WordCount program in C++. The compilation works fine, but when I tried to run my program, an error appeared :
bin/hadoop pipes -conf ../dev/word.xml -input testtile.txt -output wordcount-out
11/06/06 14:23:40 WARN mapred.JobClient: No job jar file set. User classes may not be
found. See JobConf(Class) or JobConf#setJar(String).
11/06/06 14:23:40 INFO mapred.FileInputFormat: Total input paths to process : 1
11/06/06 14:23:40 INFO mapred.JobClient: Running job: job_201106061207_0007
11/06/06 14:23:41 INFO mapred.JobClient: map 0% reduce 0%
11/06/06 14:23:53 INFO mapred.JobClient: Task Id : attempt_201106061207_0007_m_000000_0, Status : FAILED
java.io.IOException
at org.apache.hadoop.mapred.pipes.OutputHandler.waitForAuthentication(OutputHandler.java:188) at org.apache.hadoop.mapred.pipes.Application.waitForAuthentication(Application.java:194) at org.apache.hadoop.mapred.pipes.Application.(Application.java:149) at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:68) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253)
attempt_201106061207_0007_m_000000_0: Server failed to authenticate. Exiting
I am running Hadoop on Fedora on two nodes, and I followed instructions for configurations from that link : Running Hadoop on multi-node cluster. I tried the wordcount example of Hadoop with that command :
bin/hadoop jar hadoop-examples-0.20.203.0.jar wordcount testtile.txt wordcount-out
And that command works fine. That is why I don't understand why my program did not work. So I hope that someone have an idea about what I am doing wrong, or if someone had already resolve this error.
I do not know if I have to answer to my question in this way, or edit my question. Anyway I find the solution and I just want to tell it for everyone who will get the same error.
After few days of research and try, I understand that Fedora and C++ on 64bits for Hadoop is not a good match. I tried to compile the Hadoop wordcount C++ with ant like explained in the wiki. But ant gets me some error about : libssl and stdint.
First, if you are on Fedora you have to add -lcrypto to the LIBS variables in the .configure. That is cause the dependency on libcrypto must now be explicitely stated on these platform when linking to libssl.(see bug on Fedora).
Second issue : ant produces a lot of error about C++ files : to resolve that you just have to add an include : stdint.h on the top of the file.
Then the build success. I tried then to run wordcount example on my Hadoop cluster and it works, while mine did not. I expected that issue come from the library that I just corrected and I was right : I tried to run Hadoop example with library from the hadoop install directory and it did not work and I get the same error.
That could be explained by the fact that ant recompile the C++ library needed for Hadoop(with correction that I did) and used it, instead library provides in the Hadoop install directory.
这篇关于Wordcount C ++ Hadoop管道不起作用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
