在Spark Java应用程序中使用log4j2 [英] Using log4j2 in Spark java application

问题描述

我正在尝试在Spark作业中使用log4j 2 记录器.基本要求:log4j2配置位于类路径之外，因此我需要显式指定其位置.当我不使用spark-submit直接在IDE中运行代码时， log4j2 效果很好.但是，当我使用spark-submit将相同的代码提交给Spark集群时，它无法找到log42配置，并回退到默认的旧log4j.

I'm trying to use log4j2 logger in my Spark job. Essential requirement: log4j2 config is located outside classpath, so I need to specify its location explicitly. When I run my code directly within IDE without using spark-submit, log4j2 works well. However when I submit the same code to Spark cluster using spark-submit, it fails to find log42 configuration and falls back to default old log4j.

发射器命令

${SPARK_HOME}/bin/spark-submit \
--class my.app.JobDriver \  
--verbose \
--master 'local[*]' \
--files "log4j2.xml" \
--conf spark.executor.extraJavaOptions="-Dlog4j.configurationFile=log4j2.xml" \
--conf spark.driver.extraJavaOptions="-Dlog4j.configurationFile=log4j2.xml" \
myapp-SNAPSHOT.jar

Maven中的Log4j2依赖项

Log4j2 dependencies in maven

<dependencies>
. . . 
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
            <version>${log4j2.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>${log4j2.version}</version>
        </dependency>
        <!-- Bridge log4j to log4j2 -->
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-1.2-api</artifactId>
            <version>${log4j2.version}</version>
        </dependency>
        <!-- Bridge slf4j to log4j2 -->
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j-impl</artifactId>
            <version>${log4j2.version}</version>
        </dependency>
 <dependencies>

有什么主意我想念的吗?

Any ideas what I could miss?

推荐答案

很明显，目前Spark中没有针对log4j2的官方支持.这是有关此主题的详细讨论: https://issues.apache.org/jira/browse /SPARK-6305

Apparently at the moment there is no official support official for log4j2 in Spark. Here is detailed discussion on the subject: https://issues.apache.org/jira/browse/SPARK-6305

从实际角度讲，这意味着:

On practical side that means:

1. 如果您有权访问Spark配置和jar，并且可以对其进行修改，则在将log4j2 jar手动添加到SPARK_CLASSPATH并向Spark提供log4j2配置文件之后，仍可以使用log4j2.

1. If you have access to Spark configs and jars and can modify them, you still can use log4j2 after manually adding log4j2 jars to SPARK_CLASSPATH, and providing log4j2 configuration file to Spark.

如果您在托管的Spark集群上运行并且无法访问Spark jar/config，那么您仍然可以使用log4j2，但是其使用将仅限于在驱动程序端执行的代码.执行者运行的任何代码部分都将使用Spark执行者记录器(旧的log4j)

If you run on managed Spark cluster and have no access to Spark jars/configs, then you still can use log4j2, however its use will be limited to the code executed at driver side. Any code part running by executors will use Spark executors logger (which is old log4j)

这篇关于在Spark Java应用程序中使用log4j2的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！