在kubernetes上进行流利的日志记录会跳过日志轮换上的日志 [英] Fluentd logging on kubernetes skips logs on log rotation

问题描述

当前，我在kubernetes中具有以下架构:

Currently, I have the following architecuture in kubernetes:

• 在吊舱中，服务和小车容器(称为logger)正在运行.
• 服务写入文件，sidecar容器读取该文件并将其写入stdout.
• 将fluentd守护程序集配置为读取输出(收集在/var/log/containers/*_logger-*.log中的文件中，该文件是到另一个文件的链接(自上次文件旋转以来的最新文件，到较旧的文件，没有链接点)
• 总是3条日志消息属于同一字段(某些字段相同)

• In a pod, a service and a sidecar container (called logger) is running.
• The services writes to a file, the sidecar container reads that file and writes it to stdout.
• A fluentd daemonset is configured to read the output (which is collected in a file in /var/log/containers/*_logger-*.log, which is a link to another file (the latest file since the last file rotation, to the older files, no link points).
• Always 3 log messages belong together (some same fields)

此配置可正常处理数千条消息.

This configration works as expected for thousands of messages.

但是，这里有问题:

我注意到，虽然所有3条消息都是由服务和sidecar容器编写的，但流利的有时仅转发属于3条消息中的对数消息1或2.

I noticed that fluentd sometimes only forwards logmessage 1 or 2 of the 3 messages that belong together, although all 3 messages are written by the service and the sidecar container.

为便于说明，假定转发1，不转发2和3.经过研究，我发现在这种情况下，消息1是日志旋转之前的最后一条消息，消息2和3在另一个文件中(自旋转以来，符号链接指向该文件，因此应该读取).

For the explaination, assume 1 is forwarded, 2 and 3 not. After some research, I found out, that in such cases, message 1 is the last message before the log rotates, message 2 and 3 are in another file (where the symbolic link points to since the rotation, and therefore should be read).

因此，看起来fluentd跳过了几行，然后在kubernetes日志轮换后继续读取新文件.

Therefore, it looks like fluentd skips some lines before continue reading at the new file after the kubernetes log rotation.

• 这是一个已知问题吗?
• 为什么流利且kubernetes的行为如此?
• 主要问题:为了接收所有日志消息，我该怎么做才能防止这种情况发生?

我正在使用docker-image fluent/fluentd-kubernetes-daemonset:v0.12.33-elasticsearch

I am using the docker-image fluent/fluentd-kubernetes-daemonset:v0.12.33-elasticsearch

如果需要更多信息，请告诉我.

If more information is required, please let me know.

推荐答案

TLDR :

理论上，这应该与最新版本的fluentd-kubernetes-daemonset一起使用.如果不是，则rotate_wait的默认值可能会由于计时问题而被覆盖，对于in_tail_container_logs配置.

In theory this should work with the latest version of fluentd-kubernetes-daemonset. If it's not the default value of rotate_wait will probably need to be overwritten for the in_tail_container_logs configuration because of timing issues.

为此，您需要创建一个自定义docker映像，该映像将覆盖kubernetes.conf文件，或将配置映射与您的自定义配置一起使用，将其安装在容器中并将FLUENT_CONF设置为主配置文件在安装的目录中.

To do so you'll need to create a custom docker image that will overwrite the kubernetes.conf file, or use a config map with your custom config, mount it in the container and set FLUENT_CONF to the main config file in the mounted directory.

说明:

泊坞窗进程正在从容器的stdout和stderr中读取. 在将流刷新到日志文件时，它还将跟踪设置的限制.达到限制后，它将开始日志轮换.

The docker process is reading from both stdout and stderr of a container. While flushing the streams to the logfile it will also keep track of the set limits. When a limit has been reached it will start the log rotation.

同时流利的看着符号链接.当符号链接更改时，fluentd的文件监视程序将被触发，以将其内部指针更新为实际日志文件，并重置pos文件中的位置，因为新创建的日志文件为空.

At the same time fluentd is watching the symlink. When the symlink changes fluentd's file watcher will get triggered to update its internal pointer to the actual log file and reset the position in the pos file because the newly created log file is empty.

使用配置参数rotate_wait，我们告诉fluentd等待设置的秒数(默认为5)，以便可以选择已刷新到文件(或即将被删除)的最后一条日志行.在继续使用新创建的日志文件之前.这还将确保以正确的顺序处理日志行.

Using the config parameter rotate_wait we're telling fluentd to wait for the set amount of seconds (defaults to 5) so the last log lines that have been flushed to the file (or are soon to be) can be picked up before we're continuing with the newly created log file. This will also make sure that the log lines are processed in the correct order.

这篇关于在kubernetes上进行流利的日志记录会跳过日志轮换上的日志的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！