如何登录使用的纱线运行星火应用程序中log4j的本地文件系统？ [英] How to log using log4j to local file system inside a Spark application that runs on YARN?

问题描述

我要建一个Apache星火流媒体应用程序，不能让它记录到文件在本地文件系统的纱线运行时，它。如何才能做到这一点？

我已经设置 log4.properties 文件，以便它可以成功地写入到 / tmp目录日志文件

：本地文件系统（下面部分示出）上的目录

  log4j.appender.file = org.apache.log4j.FileAppender
log4j.appender.file.File =的/ tmp / application.log
log4j.appender.file.append = FALSE
log4j.appender.file.layout = org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern =％d个{YYYY-MM-DD HH：MM：SS}％-5p％C {1}：％L  - ％M％N
 

当我使用以下命令来运行我的星火本地应用程序：

 火花提交--class myModule.myClass --master本地[2] --deploy模式客户端myApp.jar
 

它运行正常，我可以看到日志消息将写入 /tmp/application.log 我的本地文件系统上。

但是当我通过纱线运行同一应用程序，例如

 火花提交--class myModule.myClass --master纱的客户端--nameMyModule的--total执行人-核心1 --executor-1G内存myApp.jar
 

或

 火花提交--class myModule.myClass --master纱线集群--nameMyModule的--total执行人-核心1 --executor-1G内存myApp.jar
 

我看不到任何 /tmp/application.log 机器的本地文件系统运行YARN上。

我在想什么。

解决方案

它看起来像你需要添加到启动您的任务/作业时使用的JVM参数。

尝试编辑 CONF /火花defaults.conf 为<一个href=\"https://mail-archives.apache.org/mod_mbox/incubator-spark-user/201406.mbox/[e-mail]%3E\">described这里

<$p$p><$c$c>spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/apps/spark-1.2.0/conf/log4j.propertiesspark.driver.extraJavaOptions = -Dlog4j.configuration =文件：/apps/spark-1.2.0/conf/log4j.properties

或者尝试编辑的conf / spark-env.sh 为<一个href=\"http://apache-spark-user-list.1001560.n3.nabble.com/Specifying-Spark-Executor-Java-options-using-Spark-Submit-td15015.html\">described这里添加同一个JVM的说法，虽然在CONF /火花defaults.conf条目应该工作。

如果你仍然没有得到任何快乐，你可以明确地传递你的log4j.properties的位置与你的火花提交这样，如果沿在命令行上的文件该文件包含您的JAR文件中，并在类路径的根目录

 火花提交--class sparky.MyApp --master火花：//my.host.com：7077 --confspark.executor.extraJavaOptions = -Dlog4j.configuration = log4j-executor.propertiesmyapp.jar
 

如果该文件是不是在你的类路径中使用文件： preFIX像这样完整路径

 火花提交... --conf \"spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/apps/spark-1.2.0/conf/log4j-executor.properties\" ...
 

I'm building an Apache Spark Streaming application and cannot make it log to a file on the local filesystem when running it on YARN. How can achieve this?

I've set log4.properties file so that it can successfully write to a log file in /tmp directory on the local file system (shown below partially):

log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.File=/tmp/application.log
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

When I run my Spark application locally by using the following command:

spark-submit --class myModule.myClass --master local[2] --deploy-mode client myApp.jar

It runs fine and I can see that log messages are written to /tmp/application.log on my local file system.

But when I run the same application via YARN, e.g.

spark-submit --class myModule.myClass --master yarn-client  --name "myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar

or

spark-submit --class myModule.myClass --master yarn-cluster  --name "myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar

I cannot see any /tmp/application.log on the local file system of the machine that runs YARN.

What am I missing.

解决方案

[Edited to avoid confusion]

It looks like you'll need to append to the JVM arguments used when launching your tasks/jobs.

Try editing conf/spark-defaults.conf as described here

spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/apps/spark-1.2.0/conf/log4j.properties

spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/apps/spark-1.2.0/conf/log4j.properties

Alternatively try editing conf/spark-env.sh as described here to add the same JVM argument, although the entries in conf/spark-defaults.conf should work.

If you are still not getting any joy, you can explicitly pass the location of your log4j.properties file on the command line along with your spark-submit like this if the file is contained within your JAR file and in the root directory of your classpath

spark-submit --class sparky.MyApp --master spark://my.host.com:7077 --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j-executor.properties" myapp.jar

If the file is not on your classpath use the file: prefix and full path like this

spark-submit ... --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/apps/spark-1.2.0/conf/log4j-executor.properties" ...

这篇关于如何登录使用的纱线运行星火应用程序中log4j的本地文件系统？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！