Kubernetes TCP运行状况检查 [英] Kubernetes TCP Health Check

问题描述

我正在构建.NET核心控制台微服务，并且架构师建议使用TCP健康检查而不是HTTP健康检查.因此，为了实现TCP运行状况检查，请找到我在OCP文件(deploymentconfig部分)中添加的以下配置. Jenkins构建成功，并且部署配置成功部署.

I am building .NET core Console microservice and it has been suggested by Architect to use TCP Health Check instead of HTTP Health Check. Hence in order to implement TCP health Check, please find the below configuration that I have added in the OCP file(deploymentconfig section). Jenkins build was successful and also the deployment config roll out was successful.

查询:

1. 如何确保探针正常工作?是否有办法通过TCP Health Check定期检查准备情况和活跃性探针?

2. 是否有任何语法可以使用TCP运行状况检查显式检查容器运行状况.

1. How to ensure that probes are working properly.Is there a way to verify readiness and liveliness probes is being done in regular interval with the TCP Health Check ?

2. Is there any syntax by which I can explicitly check the Container Health Status using TCP Health Check.

    readinessProbe:
      tcpSocket:
          port: 8080
      initialDelaySeconds: "${{READINESS_DELAY}}"
      periodSeconds: "${{READINESS_TIMEOUT}}"
      timeoutSeconds: "${{READINESS_TIMEOUT}}"
    livenessProbe:
      tcpSocket:
          port: 8080
      initialDelaySeconds: "${{LIVENESS_DELAY}}"
      periodSeconds: "${{LIVENESS_TIMEOUT}}"
      timeoutSeconds: "${{LIVENESS_TIMEOUT}}"

推荐答案

探针的输出保存到

Output of the probe is saved to kubelet component on each node.

在Kubernetes中作为默认设置，您可以检查Probes是否描述pod.例如，对于Pod，一切正常运行，您将找不到与此有关的任何信息.它仅显示相关事件，例如Unhealthy或Killing等.

As default in Kubernetes you can check Probes if you will describe pod. For example for pod where everything workings correctly you will not find any information about this. It shows only relevant events like Unhealthy or Killing, etc.

要检查此容器是否使用任何LivenessProbe或ReadinessProbe，您需要describe pod并找到:Containers.<containerName>.Liveness和Containers.<containerName>.Readiness.

To check if this container is using any LivenessProbe or ReadinessProbe you need to describe pod and find: Containers.<containerName>.Liveness and Containers.<containerName>.Readiness.

以下基于

Below example based on docs, but with additional changes which will guarantee it will fail.

添加:

livenessProbe:
      exec:
        command:
        - cat
        - /tmp/healthy

输出:

$ kubectl describe pod goproxy-fail
...
Events:
  Type     Reason     Age                  From                 Message
  ----     ------     ----                 ----                 -------
  Normal   Scheduled  3m38s                default-scheduler    Successfully assigned default/goproxy-fail to kubeadm-16
  Normal   Pulled     26s (x4 over 3m37s)  kubelet, kubeadm-16  Container image "k8s.gcr.io/goproxy:0.1" already present on machine
  Normal   Created    26s (x4 over 3m37s)  kubelet, kubeadm-16  Created container goproxy
  Normal   Killing    26s (x3 over 2m26s)  kubelet, kubeadm-16  Container goproxy failed liveness probe, will be restarted
  Normal   Started    25s (x4 over 3m36s)  kubelet, kubeadm-16  Started container goproxy
  Warning  Unhealthy  6s (x10 over 3m6s)   kubelet, kubeadm-16  Liveness probe failed: OCI runtime exec failed: exec failed: container_linux.go:346: starting container process caused "exec: \"cat\": executable file not found in $PATH": unknown

这意味着超过3分6秒有10项检查全部失败，状态为Unhealthy.

It means over 3m6s there was 10 checks which all failed and status is Unhealthy.

另一个默认选项是使用$ kubectl get events使用事件.输出将是微笑的，但它将收集集群中的所有事件.您可以指定namespace等

Another default option is to use events using $ kubectl get events. Output will be smilar but it will gather all events from cluster. You can specify namespace etc.

如何检查成功的探针

在默认设置下，任何地方都不会记录成功探测的输出.您需要将Kubelet日志记录级别--verbosity更改为至少调试模式(4).

Output of Successful probes is not recorded anywhere in default settings. You will need change Kubelet logging level, --verbosity to at least debugging mode (4).

要做到这一点，您必须:

To do it, you have to:

• ssh为主节点
• 编辑文件/var/lib/kubelet/kubeadm-flags.env(在ubuntu上，您需要sudo权限才能执行$ sudo su). 默认输出看起来像KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.conf"，您必须在最后添加--v=4.
  根据您的未来需求，您可以采用更高的日志级别.可以在此处中找到更多信息.
  kubeadm-flags.env中的所需值如下所示:
  KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.conf --v=4"
• 此后，您需要重新启动kubelet才能应用此新的日志记录级别.您可以使用sudo systemctl restart kubelet.

• ssh to master node
• edit file /var/lib/kubelet/kubeadm-flags.env (on ubuntu you need sudo rights to do it $ sudo su). Default output looks like KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.conf" you have to add --v=4 at the end.
  Depends on your future need you can take higher log level. More information can be found here.
  Desired value in kubeadm-flags.env would look like below:
  KUBELET_KUBEADM_ARGS="--cgroup-driver=cgroupfs --network-plugin=cni --pod-infra-container-image=k8s.gcr.io/pause:3.1 --resolv-conf=/run/systemd/resolve/resolv.conf --v=4"
• After that you need to restart kubelet to apply this new logging level. You can do it using sudo systemctl restart kubelet.

下一步是使用journactl检查kubelet日志.

Next step is to check kubelet logs using journactl.

$ journalctl -u kubelet您也可以grep它，因为当--v设置为4时，您将获得大量日志.我创建了另一个具有类似配置但具有容器名称tetest和容器名称goproxy的容器，更容易找到.

$ journalctl -u kubelet you can also grep it as you will get huge amount of logs as --v is set to 4. I have created another pod with similar config but with pod name tetest and container name goproxy, easier to find.

$ journalctl -u kubelet | grep tetest
...
Dec 31 10:29:46 kubeadm-16 kubelet[17767]: I1231 10:29:46.303112   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:29:55 kubeadm-16 kubelet[17767]: I1231 10:29:55.289330   17767 prober.go:129] Liveness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:29:56 kubeadm-16 kubelet[17767]: I1231 10:29:56.303326   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:06 kubeadm-16 kubelet[17767]: I1231 10:30:06.302931   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:15 kubeadm-16 kubelet[17767]: I1231 10:30:15.289462   17767 prober.go:129] Liveness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:16 kubeadm-16 kubelet[17767]: I1231 10:30:16.303267   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:26 kubeadm-16 kubelet[17767]: I1231 10:30:26.303248   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:35 kubeadm-16 kubelet[17767]: I1231 10:30:35.289164   17767 prober.go:129] Liveness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:36 kubeadm-16 kubelet[17767]: I1231 10:30:36.303071   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:46 kubeadm-16 kubelet[17767]: I1231 10:30:46.303751   17767 prober.go:129] Readiness probe for "tetest_default(a518f558-9b08-4ce8-86a2-81875f205826):goproxy" succeeded
Dec 31 10:30:49 kubeadm-16 kubelet[17767]: I1231 10:30:49.237565   17767 kubelet.go:1965] SyncLoop (SYNC): 1 pods; tetest_default(a518f558-9b08-4ce8-86a2-81875f205826)
...

在Kubernetes 1.16.3，操作系统Ubuntu 18.04上进行了测试.

Tested on Kubernetes 1.16.3, OS Ubuntu 18.04.

希望这会有所帮助

这篇关于Kubernetes TCP运行状况检查的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！