成功后如何保留 PIG 作业日志文件. [英] How to keep PIG job log file when it is successful.

问题描述

我注意到当运行 PIG 脚本时出现错误时，会生成并保留日志.但是当没有错误时，日志文件被删除.即使作业成功，有没有办法保留日志文件?

I noticed that when there is an error in running a PIG script, a log is generated and kept. But when there is no error, the log file is removed. Is there a way to keep the log file even when the job is successful?

推荐答案

默认错误(例如:脚本解析错误)记录到 pig.logfile 中，可以在 $PIG_HOME 中设置/conf/pig.properties.如果您也想记录状态消息，请准备一个有效的 log4j.properties 文件并将其设置在 log4jconf 属性中.

By default errors (e.g: script parsing errors) are logged to pig.logfile which can be set in $PIG_HOME/conf/pig.properties. If you want to log status messages too, then prepare a valid log4j.properties file and set it in the log4jconf property.

例如:将 $PIG_HOME/conf 中的 log4j.properties.template 重命名为 log4j.properties 并设置以下内容:

E.g: rename log4j.properties.template to log4j.properties in $PIG_HOME/conf and set the followings:

log4j.logger.org.apache.pig=info, B

# ***** A is set to be a ConsoleAppender.
#log4j.appender.A=org.apache.log4j.ConsoleAppender
# ***** A uses PatternLayout.
#log4j.appender.A.layout=org.apache.log4j.PatternLayout
#log4j.appender.A.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n

# ***** B is set to be a FileAppender.
log4j.appender.B=org.apache.log4j.FileAppender
#log4j.appender.B.File=/home/user/pig-distrib/logs/pig_success.log
log4j.appender.B.File=/home/user/pig-distrib/logs/pig.log
log4j.appender.B.layout=org.apache.log4j.PatternLayout
log4j.appender.B.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n
log4j.appender.B.Append=true

使用 Pig v0.10.0 (r1328203) 时，我发现成功的 pig 任务不会将作业的历史日志写入 hdfs 上的输出目录.
(hadoop.job.history.user.location=${mapred.output.dir}/_logs/history/)

如果您想通过各种方式获得这些历史记录，请以这种方式在您的猪脚本中设置 mapred.output.dir:

If you want to have these histories by all means then set mapred.output.dir in your pig script in this way:

set mapred.output.dir '/user/hadoop/test/output';

这篇关于成功后如何保留 PIG 作业日志文件.的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！