Hadoop Map Reduce 引用静态对象 [英] Hadoop Map Reduce reference static objects

问题描述

我的 map reduce 作业类中有一个静态对象，我想初始化一次(在 main 方法中)，然后在每个映射中对其调用一个函数.所以我有这个对象，我声明为变量的 MyObject:

I have a static object in my map reduce job class that I want to initialize once (in the main method), then call a function on it in every mapping. So I have this object, MyObject that I declare as a variable:

static MyObject obj;

在我的主要功能中，在我开始工作之前，我调用:

And in my main function, before I start the job I call:

obj = new MyObject();
obj.init();

然后在我想要调用的地图函数中:

And then in my map function I want to call:

obj.execute();

但由于某种原因，当我尝试此操作时出现空指针异常(它说 obj 为空).如果我在我的主函数中初始化它，映射器不应该将它视为已初始化吗?映射器是否看到静态变量?

But for some reason I get a null pointer exception when I try this (it says obj is null). If I initialize it in my main function, shouldn't the mapper see it as initialized? Does the mapper see static variables?

推荐答案

静态对象驻留在内存中.现在你的系统是分布式的所以你创建的对象在你的jobtracker运行的节点的内存中，而不是在其他系统上.

static object resides in memory. now your system is distributed one so object you had created is in memory of node on which your jobtracker is running not on other systems.

现在您不能将对象从作业传递到映射器，因为配置是用 xml 编写的，但是有一种解决方法，将您的对象序列化为 JSON，然后将其作为字符串放入您的配置中，并在映射器中反序列化此 json 对象

now you cannot pass object from job to mapper because config is written as xml, but there is a workaround, Serialize your object into JSON and then put it as string in your configuration and in mappers deserialize this json object

为了工作

job.getConfiguration().set("some key", "json string")

用于映射器

Configuration conf = context.getConfiguration();
conf.get("some key");

这篇关于Hadoop Map Reduce 引用静态对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！