如何使用 log4j 登录到在 YARN 上运行的 Spark 应用程序内的本地文件系统? [英] How to log using log4j to local file system inside a Spark application that runs on YARN?
问题描述
我正在构建一个 Apache Spark Streaming 应用程序,但在 YARN 上运行时无法将其记录到本地文件系统上的文件.怎样才能做到这一点?
I'm building an Apache Spark Streaming application and cannot make it log to a file on the local filesystem when running it on YARN. How can achieve this?
我已经设置了 log4.properties
文件,以便它可以成功写入本地文件系统上 /tmp
目录中的日志文件(部分如下所示):
I've set log4.properties
file so that it can successfully write to a log file in /tmp
directory on the local file system (shown below partially):
log4j.appender.file=org.apache.log4j.FileAppender
log4j.appender.file.File=/tmp/application.log
log4j.appender.file.append=false
log4j.appender.file.layout=org.apache.log4j.PatternLayout
log4j.appender.file.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n
当我使用以下命令在本地运行我的 Spark 应用程序时:
When I run my Spark application locally by using the following command:
spark-submit --class myModule.myClass --master local[2] --deploy-mode client myApp.jar
它运行良好,我可以看到日志消息已写入本地文件系统上的 /tmp/application.log
.
It runs fine and I can see that log messages are written to /tmp/application.log
on my local file system.
但是当我通过 YARN 运行相同的应用程序时,例如
But when I run the same application via YARN, e.g.
spark-submit --class myModule.myClass --master yarn-client --name "myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar
或
spark-submit --class myModule.myClass --master yarn-cluster --name "myModule" --total-executor-cores 1 --executor-memory 1g myApp.jar
我在运行 YARN 的机器的本地文件系统上看不到任何 /tmp/application.log
.
I cannot see any /tmp/application.log
on the local file system of the machine that runs YARN.
我错过了什么.
推荐答案
看起来您需要附加到启动任务/作业时使用的 JVM 参数.
It looks like you'll need to append to the JVM arguments used when launching your tasks/jobs.
尝试将 conf/spark-defaults.conf
编辑为 此处描述
spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/apps/spark-1.2.0/conf/log4j.properties
spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/apps/spark-1.2.0/conf/log4j.properties
或者尝试将 conf/spark-env.sh
编辑为 此处描述 以添加相同的 JVM 参数,尽管 conf/spark-defaults.conf 中的条目应该可以工作.
Alternatively try editing conf/spark-env.sh
as described here to add the same JVM argument, although the entries in conf/spark-defaults.conf should work.
如果您仍然没有任何乐趣,您可以在命令行上明确传递您的 log4j.properties 文件的位置以及您的 spark-submit
,如果该文件包含在您的JAR 文件和类路径的根目录中
If you are still not getting any joy, you can explicitly pass the location of your log4j.properties file on the command line along with your spark-submit
like this if the file is contained within your JAR file and in the root directory of your classpath
spark-submit --class sparky.MyApp --master spark://my.host.com:7077 --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j-executor.properties" myapp.jar
如果文件不在您的类路径中,请使用 file:
前缀和像这样的完整路径
If the file is not on your classpath use the file:
prefix and full path like this
spark-submit ... --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/apps/spark-1.2.0/conf/log4j-executor.properties" ...
这篇关于如何使用 log4j 登录到在 YARN 上运行的 Spark 应用程序内的本地文件系统?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!