在Hadoop中链接多个MapReduce作业 [英] Chaining multiple MapReduce jobs in Hadoop

问题描述

在许多使用MapReduce的现实生活中，最终的算法最终会被分成几个MapReduce步骤。

即， Map1，Reduce1，Map2，Reduce2等。

因此，您需要输入最后一次减少的输出作为下一个地图的输入。

中间数据是您（一般情况下）在管道成功完成后不想保留的内容。另外，因为这个中间数据通常是一些数据结构（如'map'或'set'），所以您不希望在编写和读取这些键值对时付出太多努力。

在Hadoop中推荐这样做的方法是什么？

是否有一个（简单）示例显示如何处理中间数据正确的方式，包括之后的清理？解析方案

我认为这个关于雅虎开发者网络的教程将帮助你： Chaining Jobs

您可以使用 JobClient.runJob（）。来自第一份工作的数据的输出路径成为您的第二份工作的输入路径。这些需要作为参数传递给您的工作，并使用适当的代码来解析它们并为作业设置参数。

我认为上述方法可能是现在较老的mapred API的做法，但它应该仍然有效。在新的mapreduce API中会有类似的方法，但我不确定它是什么。

就作业完成后删除中间数据而言，您可以执行这在你的代码中。我之前做过的方式是使用类似于：

  FileSystem.delete（Path f，boolean recursive）; 
  

其中路径是数据HDFS上的位置。您需要确保只有在没有其他工作需要时才删除此数据。

In many real-life situations where you apply MapReduce, the final algorithms end up being several MapReduce steps.

i.e. Map1 , Reduce1 , Map2 , Reduce2 , and so on.

So you have the output from the last reduce that is needed as the input for the next map.

The intermediate data is something you (in general) do not want to keep once the pipeline has been successfully completed. Also because this intermediate data is in general some data structure (like a 'map' or a 'set') you don't want to put too much effort in writing and reading these key-value pairs.

What is the recommended way of doing that in Hadoop?

Is there a (simple) example that shows how to handle this intermediate data in the correct way, including the cleanup afterward?

解决方案

I think this tutorial on Yahoo's developer network will help you with this: Chaining Jobs

You use the JobClient.runJob(). The output path of the data from the first job becomes the input path to your second job. These need to be passed in as arguments to your jobs with appropriate code to parse them and set up the parameters for the job.

I think that the above method might however be the way the now older mapred API did it, but it should still work. There will be a similar method in the new mapreduce API but i'm not sure what it is.

As far as removing intermediate data after a job has finished you can do this in your code. The way i've done it before is using something like:

FileSystem.delete(Path f, boolean recursive);

Where the path is the location on HDFS of the data. You need to make sure that you only delete this data once no other job requires it.

这篇关于在Hadoop中链接多个MapReduce作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！