Hadoop MapReduce log4j - 将消息记录到userlogs / job_ dir中的自定义文件中？ [英] Hadoop MapReduce log4j - log messages to a custom file in userlogs/job_ dir?

问题描述

对于我来说，我不清楚如何在工作级别配置Hadoop MapReduce log4j。有人可以帮我回答这些问题。

1）如何从客户端机器添加支持log4j日志记录。即我想在客户端机器上使用log4j属性文件，因此不想干扰群集中的Hadoop log4j设置。我会认为在项目/ jar中的属性文件应该就足够了，hadoop的分布式缓存应该会继续传输map-reduce jar。

2）如何记录消息发送到$ HADOOP_HOME / logs / userlogs / job_ / dir中的自定义文件。

3）将map减少任务使用log4j属性文件吗？客户端工作提供的和Hadoop集群中提供的服务？如果是，那么log4j.rootLogger会添加两个属性值？

感谢
Srivatsan Nallazhagappan

解决方案

您可以直接在您的代码中配置log4j。例如，您可以调用 PropertyConfigurator.configure（properties）; 例如在映射器/减速器设置方法中。

这是存储在hdfs中的属性的示例：

  InputStream is = fs.open（log4jPropertiesPath）; 
属性properties = new Properties（）; 
 properties.load（is）; 
 PropertyConfigurator.configure（properties）; 
  

其中fs是FileSystem对象，log4jPropertiesPath是hdfs的路径。

使用此功能，您还可以使用job_id将日志输出到目录。例如，您可以在调用PropertyConfigurator.configure（properties）之前修改我们的属性;

  Enumeration propertiesNames = properties.propertyNames（）; 
 while（propertiesNames.hasMoreElements（））{
 String propertyKey =（String）propertiesNames.nextElement（）; 
 String propertyValue = properties.getProperty（propertyKey）; 
 
 if（propertyValue.indexOf（JOB_ID_PATTERN）！= -1）{
 properties.setProperty（propertyKey，propertyValue.replace（JOB_ID_PATTERN，context.getJobID（）。toString（）））; 
} 
} 
  

Its not clear to me as how one should configure Hadoop MapReduce log4j at a job level. Can someone help me answer these questions.

1) How to add support log4j logging from a client machine. i.e I want to use log4j property file at the client machine, and hence don't want to disturb the Hadoop log4j setup in the cluster. I would think having the property file in the project/jar should suffice, and hadoop's distributed cache should do the rest transferring the map-reduce jar.

2) How to log messages to a custom file in $HADOOP_HOME/logs/userlogs/job_/ dir.

3) Will map reduce task use both the log4j property file? the one supplied by the client job and the one present in the hadoop cluster? If yes, then the log4j.rootLogger would add both the property values?

Thanks Srivatsan Nallazhagappan

解决方案

You can configure log4j directly in your code. For example you can call PropertyConfigurator.configure(properties); e.g. in mapper/reducer setup method.

This is example with properties stored on hdfs:

        InputStream is = fs.open(log4jPropertiesPath);
        Properties properties = new Properties();
        properties.load(is);
        PropertyConfigurator.configure(properties);

where fs is FileSystem object and log4jPropertiesPath is path on hdfs.

With this you can also output logs to a dir with job_id. For example you can modify our properities before calling PropertyConfigurator.configure(properties);

Enumeration propertiesNames = properties.propertyNames();
            while (propertiesNames.hasMoreElements()) {
                String propertyKey = (String) propertiesNames.nextElement();
                String propertyValue = properties.getProperty(propertyKey);

                if (propertyValue.indexOf(JOB_ID_PATTERN) != -1) {
                    properties.setProperty(propertyKey, propertyValue.replace(JOB_ID_PATTERN, context.getJobID().toString()));
                }
            }

这篇关于Hadoop MapReduce log4j - 将消息记录到userlogs / job_ dir中的自定义文件中？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！