Hadoop WordCount示例问题，我需要做一些性能调整吗? [英] Hadoop WordCount Example Problem, Do I need to do some performance tuning?

问题描述

我是 Hadoop 的新手.

最近我只是实现WordCount示例.

Recently I just make an implementation of WordCount example.

但是，当我在带有2个输入文件(仅9个字)的单个节点上运行此程序时，这样做花费了将近33秒！！！太疯狂了，这让我很困惑!!!

But when I run this programs on my single node with 2 input files , just 9 word, it cost nearly 33 second to do such !!! so crazy, and it makes me so confusing !!!

任何人都可以告诉我这是正常现象吗?

Can any one tell me is this normal or some???

如何解决此问题?记住，我只是创建了2个输入文件，其中有9个字.

How can I fix this problem? Remember, I just create 2 input files with 9 word in it.

提交主机地址:127.0.0.1
作业ACL:允许所有用户
职位设置:成功
状态:成功
开始于:2011年8月5日星期五CST 2011年
在2011年8月5日星期五星期五14:27:53完成 完成时间:30秒

Submit Host Address: 127.0.0.1
Job-ACLs: All users are allowed
Job Setup: Successful
Status: Succeeded
Started at: Fri Aug 05 14:27:22 CST 2011
Finished at: Fri Aug 05 14:27:53 CST 2011
Finished in: 30sec

推荐答案

Hadoop对于非常小的作业而言效率不高，因为JVM启动，进程初始化等需要花费更多时间.不过，可以通过启用JVM重用在某种程度上对其进行优化.

Hadoop is not efficient for very very small jobs, as it takes more time for the JVM Startup, process initialization and others. Though, it can be optimized to some extent by enabling JVM reuse.

http://hadoop.apache. org/common/docs/r0.20.2/mapred_tutorial.html#Task + JVM + Reuse

此外，Apache Hadoop中正在进行一些工作

Also, there is some work going on this in Apache Hadoop

https://issues.apache.org/jira/browse/MAPREDUCE-1220

不确定此版本将包含在哪个版本中，或者JIRA的状态是什么.

Not sure in which release this will be included or what the state of the JIRA is.

这篇关于Hadoop WordCount示例问题，我需要做一些性能调整吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！