如何缩放max.incremental.fetch.session.cache.slots [英] How to scale max.incremental.fetch.session.cache.slots
问题描述
我正在运行一个较大的Kafka集群，但是目前我只能正确设置 max.incremental.fetch.session.cache.slots ，并且需要一些指导.与此相关的文档也不清楚: https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability
按比例，我的意思是:3个节点，〜400个主题，4500个分区，300个消费者组，500个消费者
有一阵子，我发现 FETCH_SESSION_ID_NOT_FOUND 错误出现在日志中并希望解决.
因此，我尝试增加配置中的值，重新启动所有代理，然后池迅速再次填满至其最大容量.这减少了错误的发生，但是并没有完全消除它们.起初，我将值设置为2000，它立即充满.然后分几步达到100.000.水池充满了约40分钟的时间.
从文档中，我预计当 min.incremental.fetch.session.eviction.ms 启动时，池会在2分钟后消失.>
衡量高速缓存适当大小的度量标准是什么?这些错误是否仍是我可以在经纪人上看到的可以解决的问题，还是需要寻找配置错误的消费者?如果是这样，我需要注意什么?
Fetch会话的如此高的使用率很可能是由错误的客户端引起的.
Golang客户端Sarama 出现问题，导致分配了新的Fetch会话版本1.26.0和1.26.2之间的每个提取请求，请参见 https://github.com/Shopify/sarama/pull/1644 .
我建议您检查是否有用户正在运行此客户端，并确保他们更新到最新版本.
I'm running a somewhat large Kafka cluster, but currently I'm stuck at properly setting max.incremental.fetch.session.cache.slots and would need some guidance. The documentation about this is not clear either: https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability
By scale i mean: 3 nodes, ~400 Topics, 4500 Partitions, 300 consumergroups, 500 consumers
For a while now, I'm seeing the FETCH_SESSION_ID_NOT_FOUND errors appearing in the logs and wanted to address them.
So I've tried increasing the value in the config, restarted all brokers and the pool quickly filled up again to it's max capacity. This reduced the occurrence of the errors, but they are not completely gone. At first I've set to value to 2000, it was instantly full. Then in several steps up to 100.000. And the pool was filled in ~40 Minutes.
From the documentation I was expecting the pool to cap out after 2 Minutes when min.incremental.fetch.session.eviction.ms kicks in. But this seems not to be the case.
What would be the metrics to gauge the appropriate size of the cache. Are the errors I'm still seeing anything I can fix on the brokers or do I need to hunt down misconfigured consumers? If so, what do I need to look out for?
Such a high usage of Fetch Sessions is most likely caused by a bad client.
Sarama, a Golang client, had an issue that caused a new Fetch Session to be allocated on every Fetch request between versions 1.26.0 and 1.26.2, see https://github.com/Shopify/sarama/pull/1644.
I'd recommend checking if you have users running this client and ensure they update to the latest release.
这篇关于如何缩放max.incremental.fetch.session.cache.slots的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
