Log4j RollingFileAppender不会将映射器和减速器日志添加到文件 [英] Log4j RollingFileAppender not adding mapper and reducer logs to file

问题描述

我们希望将我们的应用程序日志打印到本地节点上的文件中。我们正在使用Log4j的RollingFileAppender。

我们的 log4j.properties 档案如下：

  ODS.LOG.DIR = / var / log / appLogs 
 ODS.LOG.INFO.FILE = application.log 
 ODS.LOG.ERROR.FILE = application_error.log 
＃根记录器选项
 log4j.rootLogger =错误，控制台
 log4j.logger.com.ournamespace =错误，APP_APPENDER，ERROR_APPENDER 
 
＃
＃console 
＃如果你想要使用这个
 
 
，将console添加到rootlogger上面log4j.appender.console = org.apache.log4j.ConsoleAppender 
 log4j.appender.console.target = System.out 
 log4j.appender.console.layout = org.apache.log4j.PatternLayout 
 log4j.appender.console.layout.ConversionPattern =％d {yyyy -MM-dd HH：mm：ss}  - ％r％p％c {5}：％m％n 
 
＃将日志消息直接记录到日志文件
 log4j.appender.APP_APPENDER = org.apache.log4j.RollingFileAppender 
 log4j.appender.APP_APPENDER.File = $ {ODS.LOG.DIR} / $ {ODS.LOG.INFO.FILE} 
 log4j.appender.APP_APPENDER.MaxFileSize = 200MB 
 log4j.appender.APP_APPENDER.Max BackupIndex = 30 
 log4j.appender.APP_APPENDER.layout = org.apache.log4j.PatternLayout 
 log4j.appender.APP_APPENDER.layout.ConversionPattern =％d {yyyy-MM-dd HH：mm：ss} - ％r％p％c {10}：％m％n 
 
 
 log4j.appender.ERROR_APPENDER = org.apache.log4j.RollingFileAppender 
 log4j.appender.ERROR_APPENDER。阈值=错误
 log4j.appender.ERROR_APPENDER.File = $ {ODS.LOG.DIR} / $ {ODS.LOG.ERROR.FILE} 
 log4j.appender.ERROR_APPENDER.MaxFileSize = 200MB 
 log4j.appender.ERROR_APPENDER.MaxBackupIndex = 90 
 log4j.appender.ERROR_APPENDER.layout = org.apache.log4j.PatternLayout 
 log4j.appender.ERROR_APPENDER.layout.ConversionPattern =％d {yyyy-MM- dd HH：mm：ss}  - ％r％p％c {10}：％m％n 
  

我们的MR驱动程序日志出现在 / var / log / appLogs 目录中，但我们的映射器和reducer日志不会显示在此目录中。

我们已经使用Ambari（Hortonwork）将我们的log4j.properties代码段复制到hdfs-log4j，yarn-log4j，hbase-log4j和zookeeper-log4j数据平台）。我们的MR作业通常使用HBase输入和输出格式类。

我们需要做些什么修改以使映射器和缩减器日志出现在/ var / log / applogs目录？

编辑：日志出现在JobHistory UI syslog 但它们不会被添加到日志文件中。我们缺少什么？

解决方案

例如，要配置log4j，您可以调用 PropertyConfigurator.configure（properties）; 从你的代码例如在映射器/减速器设置方法中。

这是存储在hdfs中的属性的示例：

  InputStream is = fs.open（log4jPropertiesPath）; 
属性properties = new Properties（）; 
 properties.load（is）; 
 PropertyConfigurator.configure（properties）; 
  

其中fs是FileSystem对象，log4jPropertiesPath是hdfs的路径。

We would like our application logs to be printed to files on the local nodes. We're using Log4j's RollingFileAppender.

Our log4j.properties file is as follows:

ODS.LOG.DIR=/var/log/appLogs
ODS.LOG.INFO.FILE=application.log
ODS.LOG.ERROR.FILE=application_error.log
# Root logger option
log4j.rootLogger=ERROR, console
log4j.logger.com.ournamespace=ERROR, APP_APPENDER, ERROR_APPENDER

#
# console
# Add "console" to rootlogger above if you want to use this
#

log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.out
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}-%r %p %c{5}: %m%n

# Direct log messages to a log file
log4j.appender.APP_APPENDER=org.apache.log4j.RollingFileAppender
log4j.appender.APP_APPENDER.File=${ODS.LOG.DIR}/${ODS.LOG.INFO.FILE}
log4j.appender.APP_APPENDER.MaxFileSize=200MB
log4j.appender.APP_APPENDER.MaxBackupIndex=30
log4j.appender.APP_APPENDER.layout=org.apache.log4j.PatternLayout
log4j.appender.APP_APPENDER.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}-%r %p %c{10}: %m%n


log4j.appender.ERROR_APPENDER=org.apache.log4j.RollingFileAppender
log4j.appender.ERROR_APPENDER.Threshold=ERROR
log4j.appender.ERROR_APPENDER.File=${ODS.LOG.DIR}/${ODS.LOG.ERROR.FILE}
log4j.appender.ERROR_APPENDER.MaxFileSize=200MB
log4j.appender.ERROR_APPENDER.MaxBackupIndex=90
log4j.appender.ERROR_APPENDER.layout=org.apache.log4j.PatternLayout
log4j.appender.ERROR_APPENDER.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss}-%r %p %c{10}: %m%n

Our MR Driver logs are appearing in /var/log/appLogs directory but our mapper and reducer logs dont show up in this directory.

We have copied our log4j.properties snippet to hdfs-log4j, yarn-log4j, hbase-log4j and zookeeper-log4j using Ambari (Hortonworks Data Platform). Our MR jobs typically use HBase input and output format classes.

What changes do we need to make to have the mapper and reducer logs appear in the /var/log/appLogs directory as well?

Edit: The logs are appearing in the JobHistory UI syslog but they aren't being added to the log file. What are we missing?

解决方案

For example to configure log4j you can call PropertyConfigurator.configure(properties); from your code e.g. in mapper/reducer setup method.

This is example with properties stored on hdfs:

        InputStream is = fs.open(log4jPropertiesPath);
        Properties properties = new Properties();
        properties.load(is);
        PropertyConfigurator.configure(properties);

where fs is FileSystem object and log4jPropertiesPath is path on hdfs.

这篇关于Log4j RollingFileAppender不会将映射器和减速器日志添加到文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！