内存中的 Hadoop Reducer 值? [英] Hadoop Reducer Values in Memory?

问题描述

我正在编写一个 MapReduce 作业，该作业最终可能会在 reducer 中产生大量值.我担心所有这些值会一次加载到内存中.

I'm writing a MapReduce job that may end up with a huge number of values in the reducer. I am concerned about all of these values being loaded into memory at once.

Iterable<VALUEIN> 的底层实现是否?values 在需要时将值加载到内存中?Hadoop:权威指南似乎暗示了这种情况，但没有给出明确"的答案.

Does the underlying implementation of the Iterable<VALUEIN> values load values into memory as they are needed? Hadoop: The Definitive Guide seems to suggest this is the case, but doesn't give a "definitive" answer.

reducer 的输出将比输入的值大得多，但我相信输出会根据需要写入磁盘.

The reducer output will be far more massive than the values input, but I believe the output is written to disk as needed.

推荐答案

你读的没错.reducer 不会将所有值都存储在内存中.相反，当循环遍历 Iterable 值列表时，每个 Object 实例都会被重复使用，因此它只会在给定时间保留一个实例.

You're reading the book correctly. The reducer does not store all values in memory. Instead, when looping through the Iterable value list, each Object instance is re-used, so it only keeps one instance around at a given time.

例如在下面的代码中，objs ArrayList 在循环之后将具有预期的大小，但每个元素都将是相同的 b/c 每次迭代都会重复使用 Text val 实例.

For example in the follow code, the objs ArrayList will have the expected size after the loop but every element will be the same b/c the Text val instance is re-used every iteration.

public static class ReducerExample extends Reducer<Text, Text, Text, Text> {
public void reduce(Text key, Iterable<Text> values, Context context) {
    ArrayList<Text> objs = new ArrayList<Text>();
            for (Text val : values){
                    objs.add(val);
            }
    }
}

(如果出于某种原因您确实想对每个 val 采取进一步的措施，您应该制作一个深层副本然后存储它.)

(If for some reason you did want to take further action on each val, you should make a deep copy and then store it.)

当然，即使是单个值也可能大于内存.在这种情况下，建议开发人员采取措施减少前面 Mapper 中的数据，以使值不会太大.

Of course even a single value could be larger than memory. In this case it's recommended to the developer to take steps to pare the data down in the preceding Mapper so that the value is not so large.

更新:参见 Hadoop The Definitive Guide 2nd Edition 的第 199-200 页.

UPDATE: See pages 199-200 of Hadoop The Definitive Guide 2nd Edition.

This code snippet makes it clear that the same key and value objects are used on each 
invocation of the map() method -- only their contents are changed (by the reader's 
next() method). This can be a surprise to users, who might expect keys and vales to be 
immutable. This causes prolems when a reference to a key or value object is retained 
outside the map() method, as its value can change without warning. If you need to do 
this, make a copy of the object you want to hold on to. For example, for a Text object, 
you can use its copy constructor: new Text(value).

The situation is similar with reducers. In this case, the value object in the reducer's 
iterator are reused, so you need to copy any that you need to retain between calls to 
the iterator.

这篇关于内存中的 Hadoop Reducer 值?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！