GKE& Stackdriver:Java logback记录格式? [英] GKE & Stackdriver: Java logback logging format?

问题描述

我有一个在Kubernetes上的docker映像中运行Java的项目.流利的代理会自动提取日志，并最终将其存储在Stackdriver中.

I have a project running Java in a docker image on Kubernetes. Logs are automatically ingested by the fluentd agent and end up in Stackdriver.

但是，日志格式是错误的:多行日志在Stackdriver中放入单独的日志行，并且所有日志都具有"INFO"日志级别，即使它们确实是警告或错误.

However, the format of the logs is wrong: Multiline logs get put into separate log lines in Stackdriver, and all logs have "INFO" log level, even though they are really warning, or error.

我一直在搜索有关如何配置回传以输出正确格式以使其正常工作的信息，但是在Google Stackdriver或GKE文档中找不到此类指南.

I have been searching for information on how to configure logback to output the correct format for this to work properly, but I can find no such guide in the google Stackdriver or GKE documentation.

我的猜测是我应该以某种形式输出JSON，但是我在哪里可以找到有关格式的信息，甚至可以找到有关如何正确设置此管道的指南.

My guess is that I should be outputting JSON of some form, but where do I find information on the format, or even a guide on how to properly set up this pipeline.

谢谢！

推荐答案

此答案包含了我所需的大多数信息: https://stackoverflow .com/a/39779646

This answer contained most of the information I needed: https://stackoverflow.com/a/39779646

我调整了答案以适合我的确切问题，并修复了一些似乎已被弃用的怪异输入和代码.

I have adapted the answer to fit my exact question, and to fix some weird imports and code that seems to have been deprecated.

logback.xml:

logback.xml:

<configuration>
    <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
            <layout class="my.package.logging.GCPCloudLoggingJSONLayout">
                <pattern>%-4relative [%thread] %-5level %logger{35} - %msg</pattern>
            </layout>
        </encoder>
    </appender>
    <root level="INFO">
        <appender-ref ref="STDOUT"/>
    </root>
</configuration>

GCPCloudLoggingJSONLayout:

GCPCloudLoggingJSONLayout:

import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.PatternLayout;
import ch.qos.logback.classic.spi.ILoggingEvent;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;

import java.util.Map;

import static ch.qos.logback.classic.Level.DEBUG_INT;
import static ch.qos.logback.classic.Level.ERROR_INT;
import static ch.qos.logback.classic.Level.INFO_INT;
import static ch.qos.logback.classic.Level.TRACE_INT;
import static ch.qos.logback.classic.Level.WARN_INT;

/**
 * GKE fluentd ingestion detective work:
 * https://cloud.google.com/error-reporting/docs/formatting-error-messages#json_representation
 * http://google-cloud-python.readthedocs.io/en/latest/logging-handlers-container-engine.html
 * http://google-cloud-python.readthedocs.io/en/latest/_modules/google/cloud/logging/handlers/container_engine.html#ContainerEngineHandler.format
 * https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/logging/google/cloud/logging/handlers/_helpers.py
 * https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
 */
public class GCPCloudLoggingJSONLayout extends PatternLayout {

    private static final ObjectMapper objectMapper = new ObjectMapper();

    @Override
    public String doLayout(ILoggingEvent event) {
        String formattedMessage = super.doLayout(event);
        return doLayoutInternal(formattedMessage, event);
    }

    /**
     * For testing without having to deal wth the complexity of super.doLayout()
     * Uses formattedMessage instead of event.getMessage()
     */
    private String doLayoutInternal(String formattedMessage, ILoggingEvent event) {
        GCPCloudLoggingEvent gcpLogEvent =
            new GCPCloudLoggingEvent(formattedMessage, convertTimestampToGCPLogTimestamp(event.getTimeStamp()),
                mapLevelToGCPLevel(event.getLevel()), event.getThreadName());

        try {
            // Add a newline so that each JSON log entry is on its own line.
            // Note that it is also important that the JSON log entry does not span multiple lines.
            return objectMapper.writeValueAsString(gcpLogEvent) + "\n";
        } catch (JsonProcessingException e) {
            return "";
        }
    }

    private static GCPCloudLoggingEvent.GCPCloudLoggingTimestamp convertTimestampToGCPLogTimestamp(
        long millisSinceEpoch) {
        int nanos =
            ((int) (millisSinceEpoch % 1000)) * 1_000_000; // strip out just the milliseconds and convert to nanoseconds
        long seconds = millisSinceEpoch / 1000L; // remove the milliseconds
        return new GCPCloudLoggingEvent.GCPCloudLoggingTimestamp(seconds, nanos);
    }

    private static String mapLevelToGCPLevel(Level level) {
        switch (level.toInt()) {
            case TRACE_INT:
                return "TRACE";
            case DEBUG_INT:
                return "DEBUG";
            case INFO_INT:
                return "INFO";
            case WARN_INT:
                return "WARN";
            case ERROR_INT:
                return "ERROR";
            default:
                return null; /* This should map to no level in GCP Cloud Logging */
        }
    }

    /* Must be public for Jackson JSON conversion */
    public static class GCPCloudLoggingEvent {
        private String message;
        private GCPCloudLoggingTimestamp timestamp;
        private String thread;
        private String severity;

        public GCPCloudLoggingEvent(String message, GCPCloudLoggingTimestamp timestamp, String severity,
            String thread) {
            super();
            this.message = message;
            this.timestamp = timestamp;
            this.thread = thread;
            this.severity = severity;
        }

        public String getMessage() {
            return message;
        }

        public void setMessage(String message) {
            this.message = message;
        }

        public GCPCloudLoggingTimestamp getTimestamp() {
            return timestamp;
        }

        public void setTimestamp(GCPCloudLoggingTimestamp timestamp) {
            this.timestamp = timestamp;
        }

        public String getThread() {
            return thread;
        }

        public void setThread(String thread) {
            this.thread = thread;
        }

        public String getSeverity() {
            return severity;
        }

        public void setSeverity(String severity) {
            this.severity = severity;
        }

        /* Must be public for JSON marshalling logic */
        public static class GCPCloudLoggingTimestamp {
            private long seconds;
            private int nanos;

            public GCPCloudLoggingTimestamp(long seconds, int nanos) {
                super();
                this.seconds = seconds;
                this.nanos = nanos;
            }

            public long getSeconds() {
                return seconds;
            }

            public void setSeconds(long seconds) {
                this.seconds = seconds;
            }

            public int getNanos() {
                return nanos;
            }

            public void setNanos(int nanos) {
                this.nanos = nanos;
            }

        }
    }

    @Override
    public Map<String, String> getDefaultConverterMap() {
        return PatternLayout.defaultConverterMap;
    }
}

正如我之前所说，代码最初是从另一个答案中提取的，我只是稍微清理了一下代码以使其更适合我的用例.

As I said earlier, the code was originally from another answer, I have just cleaned up the code slightly to fit my use-case better.

这篇关于GKE& Stackdriver:Java logback记录格式?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！