使用Kubernetes中的最少连接来平衡流量 [英] Balancing traffic using least connection in Kubernetes

问题描述

我有一个Kubernetes集群，其部署类似于下一个部署:

I have a Kubernetes cluster with a deployment like the next one:

此处的目标是将应用程序部署在通过名为my-app的ClusterIP服务公开的多个Pod中.在多个名称空间(A，B和C)中进行相同的部署，略微更改了应用程序的配置.然后，在某些节​​点中，我有一个使用hostNetwork绑定到节点端口的HAProxy.这些HAProxy通过指向它们的DNS(my_app.com)向我的客户端公开.

The goal here is to deploy an application in multiple pods exposed through a ClusterIP service named my-app. The same deployment is made in multiple namespaces (A, B and C), changing slightly the config of the application. Then, in some nodes I have an HAProxy using hostNetwork to bind to the node ports. These HAProxy are exposed to my clients through a DNS pointing to them (my_app.com).

当客户端连接到我的应用程序时，他们发送一个标头，指定应将请求重定向到的名称空间(A，B或C)，并且HAProxy使用do-resolve来针对dns条目解析服务的IP my_app.A.svc.cluster.local，它返回名称空间A中服务my_app的IP.这样，我就可以为集群提供单个入口点(单个DNS记录)和单个端口(80)，这是我的要求之一.我还能够创建新的名称空间并部署应用程序的其他配置，而无需修改HAProxies，这是第二个要求.

When a client connects to my app, they send a header specifying the namespace to which the request should be redirected (A, B or C) and the HAProxy resolves the IP of the service using do-resolve against a dns entry like my_app.A.svc.cluster.local, which returns the IP of the service my_app in the namespace A. That way I can have a single entry point (single DNS record) and a single port (80) to my cluster, which is one of my requirements. I'm also able to create new namespaces and deploy other configs of my app without the need to modify the HAProxies, which is the second requirement.

现在，我收到的请求是长短请求的混合，因此我需要在这里使用最少的连接.这在HAProxies中是不可能的，因为我没有后端列表(重定向是动态的，如下面的代码所示).我正在尝试将kube-proxy与IPVS和最少连接模式一起使用.我注意到的是，跟踪到不同Pod的连接是每个节点的，并且此信息不在不同节点之间共享.这样，如果两个不同的节点处理了对my_app.com Namespace: A的两个请求，则它们都可以像在每个节点中一样都转到同一个Pod(例如pod_1)，则与该Pod的活动连接数为0.我增加了DNS背后的HAProxies的数量.

Now, the requests I get are a mix of short and long requests so I need to use least connection here. This is is not possible in the HAProxies as I don't have a list of backends (the redirection is dynamic as you can see in the code below). I'm trying to use kube-proxy with IPVS and least connection mode. What I noticed is that the tracking of connections to the different pods is per node, and this information is not shared between the different nodes. This way, if two request to my_app.com Namespace: A are processed by two different nodes, both can go to the same pod (eg. pod_1) as in each node, the number of active connections to that pod is 0. The problem becomes worse as I increase the number of HAProxies behind the DNS.

如何解决此问题并获得更好的平衡，而无需在群集中使用单个入口点(DNS后面具有单个HAProxy)?

How can I solve this problem and have a better balance without having a single entry point to the cluster (having a single HAProxy behind the DNS)?

我在这里添加了HAProxy中用于根据标头进行路由的代码:

I'm adding here the code used in HAProxy to route based on headers:

resolvers dns
    hold nx 3s
    hold other 3s
    parse-resolv-conf

frontend my_app_frontend
    bind :80
    default_backend my_app_backend
    http-request set-var(sess.namespace) hdr(X-Namespace)
    http-request do-resolve(txn.service,dns,ipv4) str(),concat(my_app.,sess.namespace,.svc.cluster.local)

backend my_app_backend
    http-request set-dst var(txn.service)
    http-request set-dst-port int(80)
    server service 0.0.0.0:0

推荐答案

我将使用HAProxy的对等功能来保存跨节点边界的名称空间的会话.
https://www.haproxy.com/blog/introduction-to -haproxy-stick-tables/

I would use the peers feature from HAProxy to save the sessions for the namespaces cross nodes border.
https://www.haproxy.com/blog/introduction-to-haproxy-stick-tables/

简而言之，未经测试

peers mypeers
  peer node1 192.168.122.64:10000
  peer node2 192.168.122.1:10000

backend my_app_backend
  stick-table type string len 32 size 100k expire 30m peers mypeers
  stick on hdr(X-Namespace)
  http-request set-dst var(txn.service)
  http-request set-dst-port int(80)
  server service 0.0.0.0:0

这篇关于使用Kubernetes中的最少连接来平衡流量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！