如何检查Kafka集群中使用的增量获取会话缓存槽的实际数量? [英] How to check the actual number of incremental fetch session cache slots used in Kafka cluster?

问题描述

我正在阅读这个问题 ​​Kafka:持续获取 FETCH_SESSION_ID_NOT_FOUND，我正在尝试应用 Hrishikesh Mishra 建议的解决方案，因为我们也面临类似的问题，所以我将代理设置 max.incremental.fetch.session.cache.slots 增加到 2000，默认为 1000.但现在我想知道如何监控已使用的增量获取会话缓存槽的实际数量，在 prometheus 中我看到 kafka_server_fetchsessioncache_numincrementalfetchpartitionscached 指标，promql 查询在三个代理中的每一个上显示的数量现在明显超过 2000，即 2703、2655 和 2054，所以如果我查看正确的指标，我会感到困惑.还有 kafka_server_fetchsessioncache_incrementalfetchsessionevictions_total 在所有代理上显示零.

I am reading this question Kafka: Continuously getting FETCH_SESSION_ID_NOT_FOUND, and I am trying to apply the solution suggested by Hrishikesh Mishra, as we also face the similar issue, so I increased the broker setting max.incremental.fetch.session.cache.slots to 2000, default was 1000. But now I wonder how can I monitor the actual number of used incremental fetch session cache slots, in prometheus I see kafka_server_fetchsessioncache_numincrementalfetchpartitionscached metrics, and promql query shows on each of three brokers the number that is now significantly over 2000, that is 2703, 2655 and 2054, so I am confused if I look at the proper metrics. There is also kafka_server_fetchsessioncache_incrementalfetchsessionevictions_total that shows zeros on all brokers.

好的，还有 kafka_server_fetchsessioncache_numincrementalfetchsessions 在三个代理中的每一个上显示 cca 500，所以总共 cca 1500，介于 1000 和 2000 之间，所以也许该指标是由 max.incremental.fetch 控制的指标.session.cache.slots ?

OK, there is also kafka_server_fetchsessioncache_numincrementalfetchsessions that shows cca 500 on each of three brokers, so that is total of cca 1500, which is between 1000 and 2000, so maybe that metrics is the one that is controlled by max.incremental.fetch.session.cache.slots ?

实际上，截至目前，每个broker上已经有700多个增量fetch session，总共2100多个，所以，很明显，每个broker都有2000的限制，所以整个数量集群可以达到 6000.现在每个 broker 上的数量低于 1000 的原因是因为在配置更改后重新启动了 broker.

Actually, as of now, it is already more than 700 incremental fetch sessions on each broker, that is total of more than 2100, so, obviously, the limit of 2000 applies to each broker, so that the number in the whole cluster can go as far as 6000. The reason why the number is now below 1000 on each broker is because the brokers were restarted after the configuration change.

问题是如何在个人消费者层面检查这种分配.这样的查询:

And the question is how can this allocation be checked on the individual consumer level. Such a query:

count by (__name__) ({__name__=~".*fetchsession.*"})

只返回这个表:

Element                                                             Value
kafka_server_fetchsessioncache_incrementalfetchsessionevictions_total{} 3
kafka_server_fetchsessioncache_numincrementalfetchpartitionscached{}    3
kafka_server_fetchsessioncache_numincrementalfetchsessions{}            3

推荐答案

名为 kafka.server:type=FetchSessionCache,name=NumIncrementalFetchSessions 的指标是监控 FetchSession 数量的正确方法.

The metric named kafka.server:type=FetchSessionCache,name=NumIncrementalFetchSessions is the correct way to monitor the number of FetchSessions.

大小可通过 max.incremental.fetch.session.cache.slots 配置.请注意，此设置适用于每个代理，因此每个代理最多可以缓存 max.incremental.fetch.session.cache.slots 个会话.

The size is configurable via max.incremental.fetch.session.cache.slots. Note that this setting is applied per-broker, so each broker can cache up to max.incremental.fetch.session.cache.slots sessions.

您看到的另一个指标 kafka.server:type=FetchSessionCache,name=NumIncrementalFetchPartitionsCached 是所有 FetchSession 中使用的分区总数.许多 FetchSession 将使用多个分区，因此预计会看到更多的分区.

The other metric you saw, kafka.server:type=FetchSessionCache,name=NumIncrementalFetchPartitionsCached, is the total number of partitions used across all FetchSession. Many FetchSessions will used several partitions so it's expected to see a larger number of them.

如您所说，您看到的 FetchSession 数量较少可能是由于重新启动所致.

As you said, the low number of FetchSessions you saw was likely due to the restart.

这篇关于如何检查Kafka集群中使用的增量获取会话缓存槽的实际数量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！