如何衡量Hadoop中每个阶段(映射，随机/排序，缩小)的持续时间? [英] How can I measure the duration of each phase(map, shuffle/sort, reduce) in Hadoop?

问题描述

Hadoop执行映射，洗牌/排序和缩减阶段. 我想知道每个阶段的持续时间. 我的意思是，它们需要多长时间?

Hadoop execute map, shuffle/sort and reduce phase. I want to know the duration of each phase. I mean, how long do they take?

我搜索了很多文档，但找不到任何明确的解决方案.例如，我设置了配置文件并启用了探查器，以便使用Hadoop探查器.但是不会生成结果文件"java.hprof.txt".此外，我不确定结果文件是否包含我想要的信息.

I have searched lots of documents but I couldn't find any clear solution. For example, I set the configure file and enable the profiler in order to use Hadoop profiler. But the result file "java.hprof.txt" is not generated. Furthermore, I'm not sure the result file includes the information what I want.

你能帮我吗?

推荐答案

跟踪工作进度的一种方法是查看Hadoop提供的Web UI.默认情况下，它在http://server-adress:50030/jobtracker.jsp处可用.

One way of tracking your job's progress is looking at the Web UI provided by Hadoop. By default it is available at http://server-adress:50030/jobtracker.jsp.

您可以在其中找到有关任务运行时，任务进度，查看日志和许多其他有用工具的信息.

There you can find information about task runtimes, task progress, view logs and many other userful tools.

这篇关于如何衡量Hadoop中每个阶段(映射，随机/排序，缩小)的持续时间?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！