如何使用码头运送在微服务架构中的日志？ [英] How to ship logs in a microservice architecture with docker?

问题描述

Heroku将其十二因素应用程序清单中的日志描述为简单的事件流：

Heroku describes logs in its Twelve-Factor App manifest as simple event streams:

日志是从所有正在运行的进程和后台服务的输出流中收集的聚合时间顺序事件流。以原始形式登录通常是每行一个事件的文本格式（尽管来自异常的回溯可能跨越多行）。日志没有固定的开始或结束，只要应用程序正在运行，流量就会持续流动。

Logs are the stream of aggregated, time-ordered events collected from the output streams of all running processes and backing services. Logs in their raw form are typically a text format with one event per line (though backtraces from exceptions may span multiple lines). Logs have no fixed beginning or end, but flow continuously as long as the app is operating.

此外，应用程序应该只是将日志写入 stdout ，将任务留给环境。

Additionally, apps should simply write logs to stdout, leaving the task to the "environment".

一个十二因素应用程序从不关心其输出流的路由或存储。它不应该尝试写入或管理日志文件。相反，每个运行的进程将其事件流（无缓冲）写入stdout。在本地开发期间，开发人员将在终端前台查看此流，观察应用程序的行为。

A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior.

在分段或生产部署中，每个进程的流将被捕获由执行环境与应用程序中的所有其他流进行整理，并路由到一个或多个最终目的地进行查看和长期归档。这些归档目的地不可见或可由应用程序配置，而是完全由执行环境管理。开源日志路由器（如Logplex和Fluent）可用于此目的。

In staging or production deploys, each process’ stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app, and instead are completely managed by the execution environment. Open-source log routers (such as Logplex and Fluent) are available for this purpose.

那么，最好的方式是实现这一点在码头环境方面的可靠性，效率和易用性？我想以下问题出现在：

So what's the best way to achieve this in a docker environment in terms of reliability, efficiency and ease of use? I think the following questions come to mind:

• 依靠Docker自己的日志设施（ docker logs

• 运行docker是否安全，并将其输出视为日志流？


• 可以 stdout 直接重定向到一个文件（磁盘空间）？


• 如果使用文件，应该在docker图像或绑定卷内（ docker run --volume = [] ）？


• 是否需要logrotation？


• 是否可以直接将stdout重定向到日志记录器（以及哪个登录器）？


• 是一个命名管道（又称FIFO）的选项？


• 更多问题？）

• Is it safe to rely on Docker's own log facility (docker logs)?
• Is it safe to run docker undetached and consider its output as the logging stream?
• Can stdout be redirected to a file directly (disk space)?
• If using a file, should it be inside the docker image or a bound volume (docker run --volume=[])?
• Is logrotation required?
• Is it safe to redirect stdout directly into a logshipper (and which logshipper)?
• Is a named pipe (aka FIFO) an option?
• (more questions?)

推荐答案

Docker 1.6 介绍 logging drivers 以提供更多的控制权日志输出。 - 日志驱动程序标志配置其中 stdout &应该指导来自运行在容器中的进程的 stderr 。另请参阅配置日志记录驱动程序。

Docker 1.6 introduced the notion of logging drivers to offer more control over log output. The --log-driver flag configures where stdout & stderr from the process running in a container should be directed. See also Configuring Logging drivers.

有几个驱动程序可用。请注意，除 json-file 之外的所有这些都禁止使用 docker日志来收集容器日志。

Several drivers are available. Note that all of these except json-file disable the use of docker logs to gather container logs.

• none - 禁用容器日志。


• json-file - 像以前一样，json格式化的stdout可用于 / var / lib / docker / containers /< containerid> /< containerid> ; -json.log


• syslog - 将消息写入syslog。还可以接受 - log-opt 通过TCP，UDP或Unix域套接字将日志消息引导到指定的syslog。也禁用 docker日志


• journald - 写入systemd日志。


• * gelf - Graylog扩展日志格式（GELF）。将日志消息写入GELF端点，如Graylog或Logstash


• * fluentd - 发送容器日志到 fluentd 。接受一些选项来自定义流畅的地址并使用日志消息发送标签。


• ** awslogs - 将日志消息写入AWS CloudWatch日志

• none - disable container logs.
• json-file - Behaves as it did previously, with json formatted stdout available in /var/lib/docker/containers/<containerid>/<containerid>-json.log
• syslog - Writes messages to syslog. Also accepts --log-opt to direct log messages to a specified syslog via TCP, UDP or Unix domain socket. Also disables docker logs
• journald - Writes to the systemd journal.
• *gelf - Graylog Extended Log Format (GELF). Writes log messages to a GELF endpoint such as Graylog or Logstash
• *fluentd - Send container logs to fluentd. Accepts some options to customize the address of the fluentd and send tags with log messages.
• **awslogs - Writes log messages to AWS CloudWatch Logs

* Docker 1.8中的新功能

** Docker 1.9中的新功能

** New in Docker 1.9

例如：

docker run --log-driver=syslog --log-opt syslog-address=tcp://10.0.0.10:1514 ...

这是Docker推荐的软件解决方案，将其日志消息写入 stdout & stderr的。但是，某些软件不会将日志消息写入 stdout / stderr 。而是写入日志文件或syslog。在这些情况下，以下原始答案的一些细节仍然适用。要重述：

This is the Docker-recommended solution for software that writes its log messages to stdout & stderr. Some software, however, does not write log messages to stdout/stderr. They instead write to log files or to syslog, for example. In those cases, some of the details from the original answer below still apply. To recap:

如果应用程序写入本地日志文件，请从主机装载卷（或使用 data-only容器到容器，并将日志消息写入该位置。

If the app writes to a local log file, mount a volume from the host (or use a data-only container to the container and write log messages to that location.

如果应用程序写入syslog，有几个选项：

If the app writes to syslog, there are several options:

• 通过挂载主机的syslog套接字（）发送到主机的syslog / dev / log ）使用 -v / dev / log：/ dev / log 。


• 如果应用程序在其配置中接受了syslog端点，请配置主机的syslog守护程序，以监听Docker网桥上的TCP和/或UDP，并使用该端点，或者发送到远程syslog主机。


• 在容器中运行一个syslog守护程序，并使用Docker链接从其他正在运行的容器访问。


• 使用 logspout 将容器日志自动路由到re mote syslog通过UDP

• Send to the host's syslog by mount the host's syslog socket (/dev/log) to the container using -v /dev/log:/dev/log.
• If the app accepts a syslog endpoint in its configuration, configure the host's syslog daemon to listen over TCP and/or UDP on the Docker bridge network, and use that endpoint. Or just send to a remote syslog host.
• Run a syslog daemon in a container, and use Docker links to access it from other running containers.
• Use logspout to automatically route container logs to a remote syslog via UDP

不要忘记容器中的任何日志应该像在主机操作系统上一样旋转。 / p>

---

Docker pre-1.6的原始答案

Don't forget that any logs within a container should be rotated just as they would on a host OS.

？p是否可以依靠Docker自己的日志设备（docker日志）？

Is it safe to rely on Docker's own log facility (docker logs)?

docker日志每次打印整个流，而不仅仅是新日志，所以不合适。 docker logs --follow 将给出 tail -f 类功能，但是你有一个docker CLI命令运行所有时间。因此，当安全运行 docker日志时，这不是最佳的。

docker logs prints the entire stream each time, not just new logs, so it's not appropriate. docker logs --follow will give tail -f-like functionality, but then you have a docker CLI command running all the time. Thus while it is safe to run docker logs, it's not optimal.

是否可以安全地运行docker undetached，并将其输出视为日志流？

Is it safe to run docker undetached and consider its output as the logging stream?

您可以使用systemd启动容器，而不是守护进程，从而捕获系统日志中的所有stdout，然后可以由主机管理不过你想要

You can start containers with systemd and not daemonize, thus capturing all the stdout in the systemd journal which can then be managed by the host however you'd like.

可以将stdout直接重定向到文件（磁盘空间）吗？

Can stdout be redirected to a file directly (disk space)?

您可以使用 docker run ...> logfile 当然是自动化和管理感觉很脆弱，更难。

You could do this with docker run ... > logfile of course, but it feels brittle and harder to automate and manage.

如果使用文件，应该它是在码头图像或绑定的卷（docker运行--volume = []）？

If using a file, should it be inside the docker image or a bound volume (docker run --volume=[])?

如果你写在容器内那么您需要在容器中运行logrotate或某些东西来管理日志文件。更好地从主机安装卷并使用主机的日志循环守护进程控制它。

If you write inside the container then you need to run logrotate or something in the container to manage the log files. Better to mount a volume from the host and control it using the host's log rotation daemon.

需要logrotation吗？

Is logrotation required?

当然，如果应用程序写入日志，则需要像本机操作系统环境一样旋转它们。但是，如果您在容器中写入日志文件位置不可预测，那么更难。如果您在主机上旋转，则日志文件将位于，例如使用devicemapper作为存储驱动程序， / var / lib / docker / devicemapper / mnt /< containerid> / rootfs / ... 。一些丑陋的包装将需要让logrotate找到该路​​径下的日志。

Sure, if the app writes logs you need to rotate them just like in a native OS environment. But it's harder if you write inside the container since the log file location isn't as predictable. If you rotate on the host, the log file would live under, for example with devicemapper as the storage driver, /var/lib/docker/devicemapper/mnt/<containerid>/rootfs/.... Some ugly wrapper would be needed to have logrotate find the logs under that path.

将stdout直接重定向到日志记录器（和哪个日志记录器）？

Is it safe to redirect stdout directly into a logshipper (and which logshipper)?

更好地使用syslog，让日志收集器处理syslog。

Better to use syslog and let the log collector deal with syslog.

一个命名管道（又名FIFO）是一个选项？

Is a named pipe (aka FIFO) an option?

命名管道不理想，因为如果管道的读取端死亡，作者（容器）将会损坏管子。即使该事件由应用程序处理，它将被阻止，直到再次读取读取器。此外，它规避码头日志。

A named pipe isn't ideal because if the reading end of the pipe dies, the writer (the container) will get a broken pipe. Even if that event is handled by the app, it will be blocked until there is a reader again. Plus it circumvents docker logs.

另请参见 post on fluentd with docker 。

请参阅Jeff Lindsay的工具 logspout ，从运行容器收集日志，并根据需要进行路由。

See Jeff Lindsay's tool logspout that collects logs from running containers and routes them however you want.

最后，请注意，容器中的stdout记录到主机上的文件 / var / lib / docker / containers /< containerid> /< containerid> -json.log 。

Finally, note that stdout from the container logs to a file on the host in /var/lib/docker/containers/<containerid>/<containerid>-json.log.

这篇关于如何使用码头运送在微服务架构中的日志？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！