Hadoop Map减少引用静态对象 [英] Hadoop Map Reduce reference static objects

问题描述

我在我的map reduce reduce类中有一个静态对象，我想初始化一次（在main方法中），然后在每个映射中调用一个函数。所以我有这个对象MyObject，我声明为一个变量：
$ b $ pre $ 静态MyObject obj;


在我的主要功能中，在我开始工作之前，我打电话给：

  obj = new MyObject（）; 
 obj.init（）; 
  

然后在我的map函数中，我想调用：

  obj.execute（）; 
  

但是由于某种原因，当我尝试这个时，我得到一个空指针异常（它表示obj为null） 。如果我初始化它在我的主要功能，不应该映射器看到它初始化？映射器是否会看到静态变量？

解决方案

静态对象驻留在内存中。
现在你的系统被分配了一个
，所以你创建的对象在你的jobtracker运行的节点的内存中，而不是其他系统上。

现在你不能将对象从作业传递给映射器，因为配置被写为xml，但有一个解决方法，将对象序列化为JSON，然后将其作为字符串放入配置中，并映射到反序列化此json对象。 b

作业

  job.getConfiguration（）。set（some key，json string ）
  

for mapper

  Configuration conf = context.getConfiguration（）; 
 conf.get（一些关键）; 
  

I have a static object in my map reduce job class that I want to initialize once (in the main method), then call a function on it in every mapping. So I have this object, MyObject that I declare as a variable:

static MyObject obj;

And in my main function, before I start the job I call:

obj = new MyObject();
obj.init();

And then in my map function I want to call:

obj.execute();

But for some reason I get a null pointer exception when I try this (it says obj is null). If I initialize it in my main function, shouldn't the mapper see it as initialized? Does the mapper see static variables?

解决方案

static object resides in memory. now your system is distributed one so object you had created is in memory of node on which your jobtracker is running not on other systems.

now you cannot pass object from job to mapper because config is written as xml, but there is a workaround, Serialize your object into JSON and then put it as string in your configuration and in mappers deserialize this json object

for job

job.getConfiguration().set("some key", "json string")

for mapper

Configuration conf = context.getConfiguration();
conf.get("some key");

这篇关于Hadoop Map减少引用静态对象的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！