在oozie中执行时，为什么我的应用程序级别日志消失了? [英] Why do my application level logs disappear when executed in oozie?

问题描述

我在CDH5环境中使用oozie.我也在使用oozie网络控制台.我无法从应用程序中看到任何日志.我可以看到hadoop日志，spark日志等；但我看不到任何应用程序专用日志.

I'm using oozie in CDH5 environment. I'm also using the oozie web-console. I'm not able to see any of the logs from my application. I can see hadoop logs, spark logs, etc; but I see no application specific logs.

在我的应用程序中，我包含了src/main/resources/log4j.properties

In my application I've included src/main/resources/log4j.properties

# Root logger option
log4j.rootLogger=INFO, stdout

# Direct log messages to stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

在我的oozie工作流程中，我有Java动作和Spark动作.

In my oozie workflow I have java-actions and spark-actions.

还必须注意，当我从命令行运行应用程序时，确实会看到我的应用程序级别日志.

It is also important to note that when I run my application from the command line I do see my application level logs.

推荐答案

Oozie在不同的启动器"作业中运行每个Action-实际上是一个具有单个映射器的YARN作业(请参见下面的例外情况)

Oozie runs each Action in a different "launcher" job -- actually a YARN job with a single mapper (see exceptions below).

只要您看到格式为job_000000000_0000的外部ID"，就可以访问application_000000_0000 的YARN日志(是的，"job"是Hadoop 1中的旧命名约定，仍由JobHistory使用)服务，但YARN还有另一种命名约定).

Whenever you see an "external ID" in the form job_000000000_0000 then you can reach the YARN logs for application_000000_0000 (yeah, "job" is the legacy naming convention from Hadoop 1, still used by JobHistory service, but YARN has another naming convention).

您的应用程序输出实际上已转储到该Oozie启动程序"的YARN日志中

Your application output is actually dumped into the YARN logs for that Oozie "launcher"

• 您的StdErr是按原样转储的，可以在"stderr"部分中检索
• 您的StdOut将在每行(Oozie使用该前缀管理Shell和Pig动作的<capture_output/>技巧)末尾带有前缀的地方转储部分
• 没有任何东西进入AFAIK的"syslog"部分

• your StdErr is dumped as-is and can be retrieved in the "stderr" section
• your StdOut is dumped with a prefix on each line (that prefix is used by Oozie to manage its <capture_output/> trick for Shell and Pig actions) at the end of the atrocely verbose "stdout" section
• and nothing gets into the "syslog" section AFAIK

底线:

1. 运行oozie job -info ******以获得操作列表以及用于Oozie工作流程执行的相应外部ID"
2. 对于每个job_*****_**旧版ID，运行yarn logs -applicationId application_*****_** | more来浏览全局YARN日志，然后放大您的特定应用程序日志
3. 现在您可以尝试使事情自动化...玩得开心

1. run oozie job -info ****** to get the list of Actions and the corresponding "external IDs" for your Oozie workflow execution
2. for each job_*****_** legacy ID, run yarn logs -applicationId application_*****_** | more to skim the global YARN logs, then zoom on your specific app logs
3. now you can try to automate that thing... have fun           B-)

---

启动器" Oozie工作原理的例外情况-电子邮件操作/文件系统操作只是直接从Oozie服务器进程执行的API调用；然后MapReduce操作会生成带有多个Mappers和Reducers的常规YARN作业.

---

Exceptions to the "launcher" Oozie job principle -- the E-mail Action / Filesystem Action are just API calls executed directly from the Oozie server process; and the MapReduce Action spawns a regular YARN job with multiple Mappers and Reducers.

这篇关于在oozie中执行时，为什么我的应用程序级别日志消失了?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！