Kubernetes Nginx:如何实现零停机部署? [英] Kubernetes Nginx: How to have zero-downtime deployments?

问题描述

我正在尝试以零停机时间进行 kubernetes nginx 部署.该过程的一部分是启动滚动更新，以确保至少有一个 pod 始终在运行 nginx.这非常有效.

I am attempting to have a kubernetes nginx deployment with zero downtime. Part of that process has been to initiate a rollingUpdate, which ensures that at least one pod is running nginx at all times. This works perfectly well.

当旧的 nginx pod 终止时，我遇到了错误.根据关于 termination 的 kubernetes 文档，kubernetes 将:

I am running into errors when the old nginx pod is terminating. According to the kubernetes docs on termination, kubernetes will:

1. 从服务的端点列表中删除 pod，所以它是终止开始时未收到任何新流量
2. 如果已定义则调用 pre-stop 钩子，并等待它完成
3. 向所有剩余进程发送 SIGTERM
4. 在宽限期到期后向所有剩余进程发送 SIGKILL.

我知道命令 nginx -s quit 应该通过等待所有 worker 在 master 终止之前完成请求来优雅地终止 nginx.它优雅地响应 SIGQUIT 命令，而 SIGTERM 导致暴力终止.其他论坛说，只需在部署中添加以下 preStop 挂钩即可:

I understand that the command nginx -s quit is supposed to gracefully terminate nginx by waiting for all workers to complete requests before the master terminates. It responds gracefully to the SIGQUIT command, while SIGTERM results in violent termination. Other forums say that it is as easy as adding the following preStop hook to your deployment:

lifecycle:
  preStop:
    exec:
      command: ["/usr/sbin/nginx", "-s", "quit"]

然而，通过测试这个命令，我发现 nginx -s quit 立即返回，而不是等待工作人员完成.它也不返回主进程的 PID，这正是我所希望的 D:

However, from testing this command I have found that nginx -s quit returns immediately, instead of waiting for the workers to complete. It also does not return the PID of the master process, which is what I was hoping for D:

发生的情况是，kubernetes 调用 nginx -s quit，它将向工作子进程发送一个正确的 SIGQUIT，但不会等待它们完成.相反，它会直接跳到第 3 步并 SIGTERM 这些进程，导致暴力终止，从而失去连接.

What happens is, kubernetes invokes nginx -s quit, which will send a proper SIGQUIT to the worker children, but not wait for them to complete. Instead it will jump right to step 3 and SIGTERM those processes instead, resulting in violent termination, and thus, lost connections.

问题:有没有人想出一种在滚动部署期间优雅地关闭 nginx 控制器并实现零停机的好方法?sleep 解决方法不够好，我正在寻找更强大的方法.

QUESTION: Has anyone figured out a good way to gracefully shut down their nginx controller during a rolling deployment and have zero downtime? A sleep workaround isn't good enough, I'm looking for something more robust.

以下是完整的部署yaml:

Below is the full deployment yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: nginx-ingress-controller
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
 template:
    metadata:
      labels:
        app: nginx-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      serviceAccount: nginx
      containers:
        - name: nginx-ingress-controller
          image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.8
          imagePullPolicy: Always
          readinessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
          livenessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 5
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-backend
            - --v=2
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - containerPort: 80
          lifecycle:
            preStop:
              exec:
                command: ["/usr/sbin/nginx", "-s", "quit"]

推荐答案

我讨厌回答我自己的问题，但经过一番思考后，这就是我目前所拥有的.

I hate answering my own questions, but after noodling a bit this is what i have so far.

我创建了一个半阻塞的 bash 脚本，称为 killer:

I created a bash script that is semi-blocking, called killer:

#!/bin/bash

sleep 3
PID=$(cat /run/nginx.pid)
nginx -s quit

while [ -d /proc/$PID ]; do
  sleep 0.1
done

我发现在 nginx pod 中有一个文件 /run/nginx.pid，其中包含主进程的 PID.如果你调用 nginx -s quit 并开始等待直到进程消失，你实际上已经使退出命令阻塞"了.

I found that inside the nginx pod there is a file /run/nginx.pid which has the PID of the master process. If you call nginx -s quit and initiate a wait until the process disappears, you have essentially made the quit command "blocking".

请注意，在任何事情发生之前都有一个 sleep 3.这是由于竞争条件，Kubernetes 将 pod 标记为终止，但需要一点时间(<1s)从将流量指向它的服务中删除此 pod.

Note that there is a sleep 3 before anything happens. This is due to a race condition where Kubernetes marks a pod as terminating, but takes a little time (< 1s) to remove this pod from the service that points traffic toward it.

我已将此脚本安装到我的 pod 中，并通过 preStop 指令调用它.它大多有效，但在测试期间仍然偶尔会出现错误，我收到一个卷曲错误，即连接被对等重置".但这是朝着正确方向迈出的一步.

I have mounted this script into my pod, and called it via the preStop directive. It mostly works, but during testing there are still occasional blips where i get a curl error that the connection was "reset by peer." But this is a step in the right direction.

这篇关于Kubernetes Nginx:如何实现零停机部署?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！