从Docker容器内部进行结构化日志记录 [英] Structured logging to journald from within a docker container

问题描述

将结构化日志从Docker容器中写入日志的最佳方法是什么？

What is the best way to write structured logs to journald from within a docker container?

例如，我有一个应用程序使用 sd_journal_send
我没有更改应用程序，而是尝试通过

For example, I have an application that writes using sd_journal_send Rather than change the app, I have tried passing through

-v / var / log / systemd / journal：/ var / log / systemd / journal

-v /var/log/systemd/journal:/var/log/systemd/journal

在我的Ubuntu 16.04桌面上，但不在运行该应用程序的CoreOS实例上（使用Ubuntu 16.04基本映像）。我不太明白为什么。也许有更好的方式发送到日记本？

It works on the my Ubuntu 16.04 desktop, but not on the CoreOS instances where the app runs (which use the Ubuntu 16.04 base image). I don't quite understand why. Perhaps there is a better way to send to the journal?

码头工人日记输出日志记录选项有哪些限制？似乎它不仅仅支持应用程序编写消息字段。

What limitations does the docker journald output logging option have? It didn't appear it supported apps writing more than just the message field.

-

所以我已经发现我需要 -v / dev / log：/ dev / log

So I have found I need -v /dev/log:/dev/log

但是还有另一个问题与启动Docker容器的服务文件没有关联。手动添加UNIT：servicename.service无法解决。因此，在查看和运送服务日志时，它与exe关联，而不与容器或服务关联。谁遇到了这些问题，您又如何解决这些问题？

But there is also another problem in that there is no association to the service file that starts the docker container. Manually adding UNIT: servicename.service didn't solve it. And so when looking at and shipping logs for the service, it's associated with the exe but not the container or service. Whose encountered these problems and how have you solved them?

-
好​​吧，让我扩大一下范围。

-- Ok let me expand this a bit.

AC程序可以这样写入系统日记：

A C program can write to the systemd journal like this:

#include <systemd/sd-journal.h>
#include <unistd.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
        sd_journal_send("MESSAGE=Hello World!",
                        "MESSAGE_ID=52fb62f99e2c49d89cfbf9d6de5e3555",
                        "PRIORITY=5",
                        "HOME=%s", getenv("HOME"),
                        "TERM=%s", getenv("TERM"),
                        "PAGE_SIZE=%li", sysconf(_SC_PAGESIZE),
                        "N_CPUS=%li", sysconf(_SC_NPROCESSORS_ONLN),
                        NULL);
        return 0;
}

这将写入日志并添加自定义字段，例如HOME，TERM，PAGE_SIZE等当我使用journalbeat将它们发送到ELK堆栈时，这些字段最终在elasticsearch中很好地结束了，我可以直接对其进行搜索。

This writes to the journal and adds custom fields like HOME, TERM, PAGE_SIZE etc. When I use journalbeat to ship them to the ELK stack, those fields end up nicely in elasticsearch and I can search on them directly.

但是，似乎docker只是简单地获取应用程序的标准输出，并仅将其添加到几个字段中，即可将其输入到日志中。例如CONTAINER_ID。

However, it seems that docker, simply takes the stdout of apps, and feeds it to the journald with only a few fields it adds itself. e.g. CONTAINER_ID.

在docker容器中使用此类程序，然后从服务文件运行它们时，会产生一个小问题。

When using programs like this inside a docker container and then running them from a service file it creates a slight problem.

1）我必须通过一些目录和设备文件才能使用sd_journal_send进行写入。

1) I have to pass through some directories and device files to get it to write with sd_journal_send.

2）如果您从systemd .service启动容器文件，并期望使用journalctl -u服务名并查看消息，但未看到这些日志消息，因为它们以不同的方式进入了日志并且与运行它们的服务没有关联。

2) if you start the container from a systemd .service file and expect to use journalctl -u servicename and see the messages, those log messages aren't seen because they went into the journal a different route and don't get associated with the service that ran them.

3）您可以使用泊坞窗的日志记录驱动程序添加一些任意字段/标签，这些字段/标签是固定的，一次添加将出现在发送的每条消息中，并且保持不变。它们不是像我上面的C代码所希望的动态字段。

3) You can add some arbitary fields/tags using docker's journald logging driver, they are fixed, one time additions that will appear on every message sent and are unchanging. They are not dynamic fields like I want from the C code above.

本质上，在我的情况下，日志记录日志驱动程序是不够的。

Essentially, the journald log driver is insuffient in my case.

关于如何同时链接服务名称的任何建议，以便journalctl -u显示来自sd_journal_send的日志消息？

Any suggestions on how both to link the service name so journalctl -u shows up the log messages from sd_journal_send? as that would fix it then.

-
我找到了一个解决方案。如果其他人对我的解决方案感兴趣，我将在下面给出答案。

-- I've found a solution. I'll put the answer below in case others are interested in how I've solved it.

推荐答案

最终的解决方案非常简单。

The eventual solution turned out to be really simple.

我改用纯json编写消息了。因此，journalctl -u现在可以工作并显示MESSAGE字段，其中现在包含json数据。

I switched to writing my messages as pure json. So journalctl -u now works and shows the MESSAGE field now containing json data.

然后，我使用journalbeat将其发送到logstash。

I then used journalbeat to send that to logstash.

我添加到logstash.conf：

To logstash.conf I added:

filter {
  json {
    source => "message"
  }
}

这是为了扩展json将消息字段中的数据发送到顶级的单独字段中，然后再将其发送到elasticsearch。

What this does is it expands the json data from the message field into separate fields at the top level before sending them to elasticsearch.

有关logstash的json过滤器的详细信息，请参见此处

Details about the json filter for logstash can be found here

这篇关于从Docker容器内部进行结构化日志记录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！