将主服务器从1.6.13-gke.0升级到1.7.11-gke.1后的日志泛洪 [英] Log flood after master upgrade from 1.6.13-gke.0 to 1.7.11-gke.1

问题描述

我们有一个包含以下内容的GKE集群:

We have a GKE cluster with:

• 主节点版本为1.6.13-gke.0
• 2个版本为1.6.11-gke.0的节点池

我们已激活Stackdriver Monitoring和Logging.

We have Stackdriver Monitoring and Logging activated.

2018年1月22日，Google将其母版升级到1.7.11-gke.1.版本.

On 2018-01-22, masters where upgraded by Google to version 1.7.11-gke.1.

升级后，我们会出现很多类似这样的错误:

After this upgrade, we have a lot of errors like these:

I  2018-01-25 11:35:23 +0000 [error]: Exception emitting record: No such file or directory @ sys_fail2 - (/var/log/fluentd-buffers/kubernetes.system.buffer..b5638802e3e04e72f.log, /var/log/fluentd-buffers/kubernetes.system.buffer..q5638802e3e04e72f.log)

I  2018-01-25 11:35:23 +0000 [warn]: emit transaction failed: error_class=Errno::ENOENT error="No such file or directory @ sys_fail2 - (/var/log/fluentd-buffers/kubernetes.system.buffer..b5638802e3e04e72f.log, /var/log/fluentd-buffers/kubernetes.system.buffer..q5638802e3e04e72f.log)" tag="docker"

I    2018-01-25 11:35:23 +0000 [warn]: suppressed same stacktrace

这些消息每天都会淹没我们的日志〜25Gb日志，并且是由DaemonSet管理的Pod(称为fluentd-gcp-v2.0.9)生成的.

Those messages are flooding our logs ~ 25Gb of logs each day, and are generated by pods managed by a DaemonSet called fluentd-gcp-v2.0.9 .

我们发现这是一个错误，已在1.8和

We found that it's a bug fixed on 1.8 and backported to 1.7.12.

我的问题是:

1. 我们应该将母版升级到1.7.12版吗?这样做安全吗?或
2. 在升级之前还有其他测试方法吗?

谢谢.

推荐答案

首先，问题2的答案.

作为替代方案，我们可以:

As alternatives we could have:

• 已过滤fluentd以忽略fluentd-gcp容器中的日志，或者
• 停用Stackdriver监视和日志记录

回答问题1:

我们在测试环境中升级到1.7.12.该过程耗时3分钟.在这段时间内，我们无法编辑集群，也无法使用kubectl(如预期的那样)对其进行访问.

We upgraded to 1.7.12 in a test environment. The process took 3 minutes. During this period of time, we could not edit our cluster nor access it with kubectl (as expected).

升级后，我们删除了所有名为 fluentd-gcp-* 的吊舱，洪水立即停止了:

After the upgrade, we deleted all our pods called fluentd-gcp-* and the flood stopped instantly:

for pod in $(kubectl get pods -nkube-system | grep fluentd-gcp | awk '{print $1}'); do \
    kubectl -nkube-system delete pod $pod; \
    sleep 20; \
done;

这篇关于将主服务器从1.6.13-gke.0升级到1.7.11-gke.1后的日志泛洪的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！