Spring-Boot 日志记录到 Kafka:如何消除警告；最佳实践 [英] Spring-Boot logging to Kafka: how to eliminate warning; best practices

问题描述

有人问我如何从作为 Java 操作从 Oozie 运行的 Spring Boot 应用程序中捕获日志输出.

I was asked to how to capture logging output from a Spring Boot application that runs as a Java action from Oozie.

我最初的想法是可以编辑一些 log4j 属性来捕获 YARN 或 Oozie 中的应用程序日志.然后我突然想到，对于在各种集群节点上运行的特定应用程序，Kafka 将是一种更容易捕获和聚合日志消息的方法.通过订阅主题来监控分布式系统比通过日志文件钓鱼要容易得多.

My initial thought was that it would be possible edit some log4j properties to capture the application logs inside YARN or Oozie. Then it occurred to me that that Kafka would be a much easier way to capture and aggregate log messages, for a specific application, running on various cluster nodes. It's much easier to monitor a distributed system by subscribing to a topic than fishing through log files.

我注意到 Kafka 有一个 log4j appender，所以我尝试创建一个最小的可重现示例(发布在 github 上:https://github.com/alexwoolford/spring-boot-log-to-kafka-example).下面是 pom.xml 的一个片段:

I notice that Kafka has a log4j appender and so I tried to create a minimal reproducible example (posted on github: https://github.com/alexwoolford/spring-boot-log-to-kafka-example). Here's a snippet from pom.xml:

<parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>1.4.4.RELEASE</version>
</parent>

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
        <exclusions>
            <exclusion>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-logging</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.springframework.boot</groupId>
                <artifactId>logback-classic</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-log4j</artifactId>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-log4j-appender</artifactId>
        <version>0.10.0.0</version>
    </dependency>
    <dependency>
        <groupId>net.logstash.log4j</groupId>
        <artifactId>jsonevent-layout</artifactId>
        <version>1.7</version>
    </dependency>
    <dependency>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
        <version>1.2</version>
    </dependency>
</dependencies>

我的 log4j.properties 文件如下所示:

My log4j.properties file looks like this:

log4j.rootLogger=INFO
log4j.appender.KAFKA=org.apache.kafka.log4jappender.KafkaLog4jAppender
log4j.appender.KAFKA.layout=net.logstash.log4j.JSONEventLayoutV1
log4j.appender.KAFKA.topic=logs
log4j.appender.KAFKA.brokerList=hdp-single-node:6667
log4j.appender.KAFKA.syncSend=true
log4j.appender.KAFKA.producer.type=async
log4j.logger.io.woolford=INFO, KAFKA

这有效，只是它会产生警告:

This works, except that it generates a warning:

log4j:WARN No appenders could be found for logger (org.apache.kafka.clients.producer.ProducerConfig).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

即使此应用程序运行并执行我需要的操作，警告表明我配置错误.你能看出需要改变什么吗?

Even though this application runs and does what I need it to, the warnings suggest that I've misconfigured something. Can you see what needs to be changed?

另外，我注意到 Spring Boot 默认使用 Logback 并且我注意到有一个开源项目，logback-kafka-appender，允许 Logback 附加到 Kafka.Kafka log4j appender 是 Spring Boot 登录到 Kafka 的最佳方式吗?

Also, I notice that Spring Boot, by default, uses Logback and I notice that there's an open source project, logback-kafka-appender, that allows Logback to append to Kafka. Is the Kafka log4j appender the best way for Spring Boot to log to Kafka?

推荐答案

Log4j2 有一个 Kafka appender.有必要将 spring-boot-starter-log4j2 和 jackson-databind 工件添加到 pom.xml:

Log4j2 has a Kafka appender. It was necessary to add the spring-boot-starter-log4j2 and jackson-databind artifacts to the pom.xml:

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
        <exclusions>
            <exclusion>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-starter-logging</artifactId>
            </exclusion>
            <exclusion>
                <groupId>org.springframework.boot</groupId>
                <artifactId>logback-classic</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-log4j2</artifactId>
    </dependency>
    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-log4j-appender</artifactId>
        <version>0.10.0.0</version>
        <exclusions>
            <exclusion>
                <groupId>org.slf4j</groupId>
                <artifactId>slf4j-log4j12</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>2.8.6</version>
    </dependency>
</dependencies>

然后我创建了一个 XML 格式的 log4j2.xml 文件:

I then created an XML formatted log4j2.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="info" name="spring-boot-log-to-kafka-example" packages="io.woolford">
    <Appenders>
        <Kafka name="kafkaAppender" topic="logs">
            <JSONLayout />
            <Property name="bootstrap.servers">hdp-single-node:6667</Property>
        </Kafka>
    </Appenders>
    <Loggers>
        <Root level="INFO">
            <AppenderRef ref="kafkaAppender"/>
        </Root>
        <Logger name="org.apache.kafka" level="WARN" />
    </Loggers>
</Configuration>

日志消息以 JSON 格式发送到 Kafka，例如

The logging messages are sent to Kafka in JSON format, e.g.

{
    "timeMillis": 1485736022854,
    "thread": "Thread-1",
    "level": "INFO",
    "loggerName": "org.springframework.context.annotation.AnnotationConfigApplicationContext",
    "message": "Closing org.springframework.context.annotation.AnnotationConfigApplicationContext@20140db9: startup date [Sun Jan 29 17:26:52 MST 2017]; root of context hierarchy",
    "endOfBatch": false,
    "loggerFqcn": "org.apache.commons.logging.impl.SLF4JLocationAwareLog",
    "threadId": 19,
    "threadPriority": 5
}

这篇关于Spring-Boot 日志记录到 Kafka:如何消除警告；最佳实践的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！