消耗Google Cloud Kubernetes容器中磁盘空间的日志上的日志轮换 [英] Log rotation on logs consuming disk space in Google Cloud Kubernetes pod

问题描述

我们在Google Cloud Platform Kubernetes集群中有一个Pod，将JsonFormatted写入StdOut.这是由Stackdriver开箱即用的.但是，我们看到Pod的磁盘使用量正在不断增长，并且我们不知道如何在Deployment上为日志轮换设置最大大小.

We have a pod in a Google Cloud Platform Kubernetes cluster writing JsonFormatted to StdOut. This is picked up by Stackdriver out of box. However, we see the disk usage of the pod just growing and growing, and we can't understand how to set a max size on the Deployment for log rotation.

关于Google Cloud和Kubernetes的文档尚不清楚.

Documentation on Google Cloud and Kubernetes is unclear on this.

这只是最后一个小时:

推荐答案

由于日志的原因，您确定Pod的磁盘使用率很高吗? 如果应用程序将日志写入stdout，则它不使用Pod内的任何磁盘空间. 所有日志通常都存储在节点文件系统上的日志文件中，并且可以由节点logrotate进程进行管理.

Are you sure that disk usage of the pod is high because of the logs? If the application writes logs to stdout, it doesn't use any disk space inside the pod. All logs are usually stored in a log file on the node’s filesystem and can be managed by the node logrotate process.

也许应用程序将pod的磁盘空间用于其他内容，例如临时文件或调试信息?

Perhaps application uses pod's disk space for something else, like temp files or debug information?

这是与日志轮换相关的文档的一部分:

Here is the part of documentation related to log rotation:

在节点级别登录:

容器化应用程序向stdout和stderr写入的所有内容都是 由容器引擎处理并重定向到某个地方.例如， Docker容器引擎会将这两个流重定向到记录 驱动程序，该驱动程序在Kubernetes中配置为写入其中的文件 json格式.

Logging at the node level:

Everything a containerized application writes to stdout and stderr is handled and redirected somewhere by a container engine. For example, the Docker container engine redirects those two streams to a logging driver, which is configured in Kubernetes to write to a file in json format.

节点级日志记录中的重要考虑因素是实现日志 轮换，这样日志就不会占用 节点.

An important consideration in node-level logging is implementing log rotation, so that logs don’t consume all available storage on the node.

Kubernetes当前不负责轮换日志，而是负责 部署工具应设置解决方案以解决该问题.为了 例如，在kube-up.sh脚本部署的Kubernetes集群中， 有一个配置为每小时运行一次的logrotate工具.

Kubernetes currently is not responsible for rotating logs, but rather a deployment tool should set up a solution to address that. For example, in Kubernetes clusters, deployed by the kube-up.sh script, there is a logrotate tool configured to run each hour.

您还可以设置容器运行时来轮换应用程序的日志 自动，例如通过使用Docker的log-opt.

You can also set up a container runtime to rotate application’s logs automatically, e.g. by using Docker’s log-opt.

在kube-up.sh脚本中，后一种方法用于COS图像上 GCP，而前一种方法则可在其他任何环境中使用.在 两种情况下，默认情况下轮换配置为在日志记录发生时进行 文件超过10MB.

In the kube-up.sh script, the latter approach is used for COS image on GCP, and the former approach is used in any other environment. In both cases, by default rotation is configured to take place when log file exceeds 10MB.

作为示例，您可以找到有关kube-up.sh的详细信息. 在相应的

As an example, you can find detailed information about how kube-up.sh sets up logging for COS image on GCP in the corresponding script.

以下是与logrotate相关的脚本的一部分:

Here is a part of the script related to logrotate:

# Installs logrotate configuration files
function setup-logrotate() {
  mkdir -p /etc/logrotate.d/
  # Configure log rotation for all logs in /var/log, which is where k8s services
  # are configured to write their log files. Whenever logrotate is ran, this
  # config will:
  # * rotate the log file if its size is > 100Mb OR if one day has elapsed
  # * save rotated logs into a gzipped timestamped backup
  # * log file timestamp (controlled by 'dateformat') includes seconds too. This
  #   ensures that logrotate can generate unique logfiles during each rotation
  #   (otherwise it skips rotation if 'maxsize' is reached multiple times in a
  #   day).
  # * keep only 5 old (rotated) logs, and will discard older logs.
  cat > /etc/logrotate.d/allvarlogs <<EOF
/var/log/*.log {
    rotate ${LOGROTATE_FILES_MAX_COUNT:-5}
    copytruncate
    missingok
    notifempty
    compress
    maxsize ${LOGROTATE_MAX_SIZE:-100M}
    daily
    dateext
    dateformat -%Y%m%d-%s
    create 0644 root root
}
EOF

}

这篇关于消耗Google Cloud Kubernetes容器中磁盘空间的日志上的日志轮换的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！