通过负载均衡器访问服务器时出现间歇性 403，不知道为什么 [英] Intermittent 403 when accessing server via Load Balancer, can't figure out why

问题描述

好的，这里是场景.我们将 API 请求发送到 NGINX 服务器，然后该服务器将其重定向到 AWS 弹性负载均衡器，该负载均衡器的目标指向我们的后端服务器.后端服务器处理请求，返回响应.没什么特别的吧?

Okay, so here's the scenario. We send our API Requests to an NGINX Server, which then redirects it to an AWS Elastic Load Balancer, which has targets pointing to our backend servers. The backend server processes the request, returns a response. Nothing out of the ordinary, right?

好吧，无论出于何种原因，有时来自特定 API 资源的 POST 请求以 403 结束.我们在代理服务器日志 (/var/log/nginx/access.log) 中看到返回了 403，然后负载均衡器日志(访问日志，写入 S3)也说 403.但是，后端服务器 (catalina.out) 中根本没有日志说请求甚至到达.这让我相信负载均衡器以某种方式丢弃了一些请求，并且从未将其发送到后端.当然，这只是表面级别的假设.我真的不确定请求在哪里卡住/丢弃.

Well, for whatever reason, sometimes the POST requests coming from a specific API resource ends up with a 403. We see it in the Proxy Server logs (/var/log/nginx/access.log) that there's a 403 returned, and then Load Balancer logs (access logs, writes to S3) also say 403. However, no logs at all in the backend servers (catalina.out) saying that the request even arrived. This makes me believe that the Load Balancer is somehow discarding some of the requests and never makes it to the backend. Of course, this is just a surface level assumption. I'm really not sure where the request is getting stuck/discarded.

需要注意的是，在 403 场景中，我们的请求返回 403 只需要 <60 毫秒.如果返回 200，通常需要大约 250 毫秒.因此，负载均衡器似乎根本没有尝试将其带到后端服务器，而只是假设某个地方出现了 403.

Something to note is that during the 403 scenario, it takes only like <60ms for our request to be returned a 403. If it returns a 200, it usually takes around ~250ms. So it seems like the Load Balancer doesn't even try to bring it to the backend server at all and just assumes a 403 somewhere.

间歇性只会使问题变得更糟，因为查明问题更加困难.

It being intermittent just makes the problem even worse, as pinpointing the problem is even harder.

我们实际上已经尝试迁移到现代应用程序负载均衡器，并且有一段时间问题逐渐缓和.但是现在，即使使用更新的负载均衡器，我们也会再次遇到更多间歇性 403.

We've actually tried migrating to a modern Application Load Balancer, and for a while the problem kind of simmered down. But now we're getting more intermittent 403s again even with the updated Load Balancer.

这个问题现在已经快一年了，仍然没有找到可以将 403 Forbidden 机会降低到接近 0% 的解决方案.

The problem's almost a year old now, and still haven't found a solution that would put the 403 Forbidden chance to near 0%.

在这里完全不知所措.任何想法将不胜感激.

Completely at a loss here. Any idea would be appreciated.

推荐答案

原来一直都是 mod_security 的错.我不知道他们是如何错过告诉我 mod_security 实际安装在后端服务器中的关键细节的，这就是请求被拦截的地方.

So it turns out that it was the fault of mod_security all this time. I don't know how I they missed telling me that crucial detail where mod_security was actually installed in the backend servers, and that's where the requests were getting intercepted.

我们最终将有关 mod_security 的一些规则列入白名单，这样它就不会主动干扰从外部来源进行的某些 API 调用.

We ended up whitelisting some rules on mod_security so that it doesn't aggressively disrupt some of the API calls being made from external sources.

这篇关于通过负载均衡器访问服务器时出现间歇性 403，不知道为什么的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！