hadoop - 映射减少任务和静态变量 [英] hadoop - map reduce task and static variable

问题描述

我刚开始从事一些 hadoop/hbase MapReduce 工作(使用 cloudera)，我有以下问题:

I just started working on some hadoop/hbase MapReduce job (using cloudera) and I have the following question :

比方说，我们有一个带有 main 和静态变量的 java 类.该类定义了对应于 Mapper 和 Reducer 任务的内部类.在开始工作之前，主要初始化静态变量.该变量在 Mapper 类中读取.然后在集群上使用hadoop jar"启动该类.

Let's say, we have a java class with a main and a static viariable. That class define inner class corresponding to the Mapper and Reducer tasks. Before lauching the job, the main initialize the static variable. This variable is read in the Mapper class. The class is then launched using 'hadoop jar' on a cluster.

我的问题:我看不到其他节点上的 Map 和 Reduce 任务如何看到该静态变量.是否有任何允许节点共享 jvm 或静态变量的hadoop 魔法"?这怎么行得通?我必须在一个班级上做这件事，我无法弄清楚这在非单节点集群中如何.谢谢

My question: I don't see how Map and Reduce tasks on other nodes can see that static variable. Is there any "hadoop magic" that allow nodes to share a jvm or static variables ? How can this even work ? I have to work on a class doing just that, and I can't figure out how this is ok in a non-mononode cluster. Thank you

推荐答案

在分布式 Hadoop 集群中，每个 Map/Reduce 任务都在自己独立的 JVM 中运行.因此，无法在不同 JVM(甚至不同节点)上运行的不同类实例之间共享静态变量.

In a distributed Hadoop cluster each Map/Reduce task runs in it's own separate JVM. So there's no way to share static variable between different class instances running on different JVMs (and even on different nodes).

但是如果你想在任务之间共享一些不可变的数据，你可以使用 配置类:

But if you want to share some immutable data between tasks, you can use Configuration class:

// driver code
Configuration config = Configuration.create();
config.setLong("foo.bar.somelong",1337);
...

// mapper code
public class SomeMapper ... {
    private long someLong = 0;
    public void setup(Context context) {
        Configuration config = context.getConfiguration();
        someLong = config.getLong("foo.bar.somelong");
    }
}

这篇关于hadoop - 映射减少任务和静态变量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！