在kubernetes中自动缩放期间某些请求失败 [英] Some requests fails during autoscaling in kubernetes

问题描述

我在microk8s上建立了一个k8s集群，并将我的应用程序移植到了它.我还添加了一个水平自动缩放器，该缩放器基于cpu负载添加了pod.自动缩放器可以正常工作，并且当负载超出目标时，它会添加Pod.当我在一段时间后移除负载时，它将杀死Pod.

I set up a k8s cluster on microk8s and I ported my application to it. I also added a horizontal auto-scaler which adds pods based on the cpu load. The auto-scaler works fine and it adds pods when there is load beyond the target and when I remove the load after some time it will kill the pods.

问题是我在同一时刻注意到自动缩放器正在创建新的pod，有些请求失败了:

The problem is I noticed at the exact same moments that the auto-scaler is creating new pods some of the requests fail:

POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  502
java.io.IOException: Server returned HTTP response code: 502 for URL: http://10.203.101.61/gateway/compile
POST Response Code :  502
java.io.IOException: Server returned HTTP response code: 502 for URL: http://10.203.101.61/gateway/compile
POST Response Code :  200
POST Response Code :  502
java.io.IOException: Server returned HTTP response code: 502 for URL: http://10.203.101.61/gateway/compile
POST Response Code :  502
java.io.IOException: Server returned HTTP response code: 502 for URL: http://10.203.101.61/gateway/compile
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200
POST Response Code :  200

我想知道这是什么原因以及如何解决?

I like to know what is the reason for this and how I can fix it?

更新: 我认为最好给我更多有关我的设置的信息:

Update: I think it is better I give you more information regarding my set up:

流量来自集群外部，但是k8s节点和生成请求的程序都在一台机器上，因此没有网络问题.有一个自定义的nginx组件，它不进行负载平衡，只是充当反向代理，并将流量发送到相应的服务.

The traffic is coming from outside of the cluster but both the k8s node and the program that generates the requests are on one machine so there is no network problem. There is a custom nginx component which doesn't do load balancing and just act as a reverse proxy and sends the traffic to respective services.

我进行了另一项测试，该测试为我提供了更多信息.我运行了相同的基准测试，但是这次没有将请求发送到反向代理(nginx)，而是使用了该特定服务的IP地址，并且在自动缩放器完成其工作并启动了多个Pod时，我没有失败的请求.我不确定是Nginx还是k8s?

I ran another test which gave me more info. I ran the same benchmarking test but this time instead of sending the requests to the reverse proxy (nginx) I used the IP address of that specific service and I had no failed request while auto-scaler did its job and launched multiple pods. I am not sure if the problem is Nginx or k8s?

推荐答案

关于您的问题:

我不确定是Nginx还是k8s?

I am not sure if the problem is Nginx or k8s?

根据ingress-nginx文档:

According to ingress-nginx docs:

NGINX入口控制器不使用服务来路由流量 到豆荚.相反，它使用Endpoints API来绕过 kube-proxy允许NGINX功能，例如会话亲和力和自定义 负载平衡算法.它还消除了一些开销，例如 iptables DNAT的conntrack条目

The NGINX ingress controller does not uses Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT

所以我认为问题出在Nginx上，它没有利用Kubernetes的所有功能(例如kube-proxy)，而是在Pod完全准备就绪之前向其发送请求.
但是显然，这个问题是大使(默认情况下)后，我遇到的问题更少将请求转发给服务(因此Kubernetes负责负载平衡并将其发送到适当的Pod).

So I believe the problem is in Nginx, which does not make use of all Kubernetes features(e.g. kube-proxy) and is sending requests to Pods before they are completely ready.
However apparently this problem was fixed in 0.23.0 (Feb/2019), so you should check your version.
Personally, I experience less issues after switching from Ingress-Nginx to Ambassador, which by default forwards requests to Services (so Kubernetes is in charge of load balancing and sending it to the proper Pod).

这篇关于在kubernetes中自动缩放期间某些请求失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！