Hadoop配置：mapred。* vs mapreduce。* [英] Hadoop configuration: mapred.* vs mapreduce.*

问题描述

我注意到有两组Hadoop配置参数：一个使用mapred。*，另一个使用mapreduce。。我猜这些可能是由于旧API与新API有关，但如果我没有弄错，这些似乎在新的API中共存。我对么？如果是这样，是否有一个普遍的语句是什么用于mapred。和mapreduce是什么。*？

解决方案

检查0.20.2的源代码，只有少数 mapreduce。* 属性，它们围绕着配置作业输入/输出格式，映射器/合并器/缩减器和分区器类（它们还向作业客户端发出信号，告知用户正在使用新的API） - 查看源代码 oahmapreduce.Job ， setUseNewAPI（）方法）

• mapreduce.inputformat.class


• mapreduce.outputformat.class
code>

• mapreduce.partitioner.class


• mapreduce.map.class


• mapreduce.combine.class
  
• mapreduce.reduce.class

它们是次级配置

输入和输出格式，无论是新的还是旧的API版本，通常都使用 mapred。* 属性

例如，地图上的信号减少了您使用 mapred.input.dir 的输入路径（无论您使用的是新的还是旧的API）。相同的输出属性 mapred.output.dir

因此，如果存在长和短是不是配置属性的实用方法（ FileInputFormat.setInputPaths（Job，String）），那么您需要检查源代码。

I noticed that there are two sets of Hadoop configuration parameters: one with mapred.* and the other with mapreduce.. I am guessing these might be due to old API vs. new API but if I am not mistaken, these seem to coexist in the new API. Am I correct? If so, is there a generalized statement what is used for mapred. and what is for mapreduce.*?

解决方案

Examining the source for 0.20.2, there are only a few mapreduce.* properties, and they revolve around configuring the job input/output format, mapper/combiner/reducer and partitioner classes (they also signal to the job client that the new API is being used by the user - look through the source for o.a.h.mapreduce.Job, setUseNewAPI() method)

• mapreduce.inputformat.class
• mapreduce.outputformat.class
• mapreduce.partitioner.class
• mapreduce.map.class
• mapreduce.combine.class
• mapreduce.reduce.class

There are some more properties but they are secondary configuration

The input and output formats, whether it be new or old API versions, typically use mapred.* properties

For example, the signal your map reduce input paths you use mapred.input.dir (whether you're using the new or old API). Same for the output property mapred.output.dir

So the long and the short of if is, if there isn't a utility method to configure the property (FileInputFormat.setInputPaths(Job, String)) then you'll need to check the source

这篇关于Hadoop配置：mapred。* vs mapreduce。*的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！