持久更改kube-env变量的推荐方法 [英] Recommended way to persistently change kube-env variables

问题描述

我们正在使用elasticsearch/kibana而不是gcp进行日志记录(基于所描述的此处).

We are using elasticsearch/kibana instead of gcp for logging (based on what is described here).

要启动fluentd-elsticsearch pod，我们已在计算实例模板"->自定义元数据"->"kube-env"中设置了LOGGING_DESTINATION=elasticsearch和ENABLE_NODE_LOGGING="true".

To have fluentd-elsticsearch pod's launched we've set LOGGING_DESTINATION=elasticsearch and ENABLE_NODE_LOGGING="true" in the "Compute Instance Template" -> "Custom metadata" -> "kube-env".

虽然在手动完成时效果很好，但是当创建具有默认值(LOGGING_DESTINATION=gcp ...)的新实例模板时，它会被每个gcloud container clusters upgrade覆盖.

While this works fine when done manually it gets overwritten with every gcloud container clusters upgrade as a new Instance Template with defaults (LOGGING_DESTINATION=gcp ...) is created.

我的问题是:如何为GKE/GCE保留这种配置?

My question is: How do I persist this kind of configuration for GKE/GCE?

我曾考虑添加 k8s-user-startup-script ，但这也在实例模板中定义，因此被gcloud container clusters upgrade覆盖. 我还尝试将 k8s-user-startup-script 添加到项目元数据，但这并未考虑在内.

I thought about adding a k8s-user-startup-script but that's also defined in the Instance Template and therefore is overwritten by gcloud container clusters upgrade. I've also tried to add a k8s-user-startup-script to the project metadata but that is not taken into account.

//编辑
用于手动切换回elasticsearch的当前解决方法(无需重新创建实例模板和实例)是

//EDIT
Current workaround (without recreating Instance Template and Instances) for manually switching back to elasticsearch is:

for node in $(kubectl get nodes -o name | cut -f2 -d/); do
    gcloud compute ssh $node \
      --command="sudo cp -a /srv/salt/fluentd-es/fluentd-es.yaml /etc/kubernetes/manifests/; sudo rm /etc/kubernetes/manifests/fluentd-gcp.yaml";
done

kubelet会捡起来，杀死fluentd-gcp并启动fluentd-es.

kubelet will pick that up, kill fluentd-gcp and start fluentd-es.

//编辑#2 现在为此运行一个启动脚本" DaemonSet:

//EDIT #2 Now running a "startup-script" DaemonSet for this:

kind: DaemonSet
apiVersion: extensions/v1beta1
metadata:
  name: startup-script
  namespace: kube-system
  labels:
    app: startup-script
spec:
  template:
    metadata:
      labels:
        app: startup-script
    spec:
      hostPID: true
      containers:
        - name: startup-script
          image: gcr.io/google-containers/startup-script:v1
          securityContext:
            privileged: true
          env:
          - name: STARTUP_SCRIPT
            value: |
              #! /bin/bash

              set -o errexit
              set -o pipefail
              set -o nounset

              # Replace Google-Cloud-Logging with EFK
              if [[ ! -f /etc/kubernetes/manifests/fluentd-es.yaml ]]; then
                if [[ -f /home/kubernetes/kube-manifests/kubernetes/fluentd-es.yaml ]]; then
                  # GCI images
                  cp -a /home/kubernetes/kube-manifests/kubernetes/fluentd-es.yaml /etc/kubernetes/manifests/
                elif [[ -f /srv/salt/fluentd-es/fluentd-es.yaml ]]; then
                  # Debian based GKE images
                  cp -a /srv/salt/fluentd-es/fluentd-es.yaml /etc/kubernetes/manifests/
                fi
                test -f /etc/kubernetes/manifests/fluentd-es.yaml && rm /etc/kubernetes/manifests/fluentd-gcp.yaml
              fi

推荐答案

没有完全支持的方法来重新配置GKE中的kube-env.正如您所发现的，您可以修改实例模板，但是不能保证该模板可以跨升级使用.

There isn't a fully supported way to reconfigure the kube-env in GKE. As you've found, you can hack the instance template, but this isn't guaranteed to work across upgrades.

另一种方法是在不启用gcp日志记录的情况下创建集群，然后创建一个DaemonSet，以将fluentd-elasticsearch pod放置在每个节点上.使用此技术，您无需编写(易碎的)启动脚本，也无需依靠内置的启动脚本在设置LOGGING_DESTINATION=elasticsearch时起作用的事实(即使未被覆盖也可能会中断升级) ).

An alternative is to create your cluster without gcp logging enabled and then create a DaemonSet that places a fluentd-elasticsearch pod on each of your nodes. Using this technique you don't need to write a (brittle) startup script or rely on the fact that the built-in startup script happens to work when setting LOGGING_DESTINATION=elasticsearch (which may break across upgrades even if it wasn't getting overwritten).

这篇关于持久更改kube-env变量的推荐方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！