监控 Netty 事件循环队列的大小 [英] Monitoring the size of the Netty event loop queues
问题描述
我们实施了对 Netty 事件循环队列的监控，以了解我们的某些 Netty 模块的问题.监视器使用 io.netty.util.concurrent.SingleThreadEventExecutor#pendingTasks 方法，该方法适用于大多数模块，但对于每秒处理几千个 HTTP 请求的模块，它似乎挂起，或者非常慢.我现在意识到文档严格指定这可能是一个问题，我觉得很蹩脚......所以我正在寻找另一种方法来实现这个监视器.
We've implemented monitoring for the Netty event loop queues in order to understand issues with some of our Netty modules.
The monitor uses the io.netty.util.concurrent.SingleThreadEventExecutor#pendingTasks method, which works for most modules, but for a module that handle a few thousand HTTP requests per second it seem to be hung, or very slow.
I now realize that the docs strictly specify this can be an issue, and I feel pretty lame... so I'm looking for another way to implement this monitor.
You can see the old code here: https://github.com/outbrain/ob1k/blob/6364187b30cab5b79d64835131d9168c754f3c09/ob1k-core/src/main/java/com/outbrain/ob1k/common/metrics/NettyQueuesGaugeBuilder.java
public static void registerQueueGauges(final MetricFactory factory, final EventLoopGroup elg, final String componentName) {
int index = 0;
for (final EventExecutor eventExecutor : elg) {
if (eventExecutor instanceof SingleThreadEventExecutor) {
final SingleThreadEventExecutor singleExecutor = (SingleThreadEventExecutor) eventExecutor;
factory.registerGauge("EventLoopGroup-" + componentName, "EventLoop-" + index, new Gauge<Integer>() {
@Override
public Integer getValue() {
return singleExecutor.pendingTasks();
}
});
index++;
}
}
}
我的问题是，有没有更好的方法来监控队列大小?
My question is, is there a better way to monitor the queue sizes?
这可能是一个非常有用的指标，因为它可用于了解延迟，也可用于在某些情况下应用背压.
This can be quite a useful metric, as it can be used to understand latency, and also to be used for applying back-pressure in some cases.
推荐答案
您可能需要在从 SingleThreadEventExecutor 实例中添加和删除任务时跟踪更改.
You'd probably need to track the changes as tasks as added and removed from the SingleThreadEventExecutor instances.
为此，您可以创建一个包装和/或扩展 SingleThreadEventExecutor 的类.然后你会有一个 java.util.concurrent.atomic.AtomicInteger 每次添加新任务时你都会调用 incrementAndGet() 并且 decrementAndGet() 每次删除/完成.
To do that you could create a class that wraps and/or extends SingleThreadEventExecutor. Then you'd have an java.util.concurrent.atomic.AtomicInteger that you'd call incrementAndGet() every time a new task is added and decrementAndGet() every time one is removed/finishes.
那个 AtomicInteger 然后会给你当前待处理任务的数量.您可能可以覆盖 pendingTasks() 以使用该值(尽管在那里要小心 - 我不是 100% 不会有副作用).
That AtomicInteger would then give you the current number of pending tasks. You could probably override pendingTasks() to use that value instead (though be careful there - I'm not 100% that wouldn't have side effects).
它会为每个正在执行的任务增加一些开销，但会使检索待处理任务的数量接近恒定速度.
It would add a bit of overhead to every task being executed, but would make retrieving the number of pending tasks near constant speed.
这样做的缺点当然是它比您目前所做的更具侵入性，因为您需要将应用配置为使用不同的事件执行器.
The downside to this is of course that it's more invasive than what you are doing at the moment, as you'd need to configure your app to use different event executors.
注意.这只是关于如何解决这个问题的建议——我没有专门用 Netty 做过这件事.虽然我过去用其他代码做过这种事情.
NB. this is just a suggestion on how to work around the issue - I've not specifically done this with Netty. Though I've done this sort of thing with other code in the past.
这篇关于监控 Netty 事件循环队列的大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
