如何在消费者组kafka中动态添加消费者 [英] How to dynamically add consumers in consumer group kafka

问题描述

我应该如何知道何时必须扩展消费者组中的消费者.有快速生产者时，消费者扩大规模的触发因素是什么?

How should I know when i have to scale the consumer in consumer group . What are the triggers for the consumers to scale when there is a fast producer ?

推荐答案

一个直接的方法是获取消费者滞后(这可以计算为提交偏移量和开始偏移量之间的差异)，如果滞后在最后一个计算n 倍增加，您可以扩大规模，反之亦然.您可能需要考虑一些边缘情况，例如，如果消费者数量下降，延迟会增加，并且自动缩放功能可能会产生更多线程/机器).

One straight forward approach would be to get the consumer lag(this can be computed as the difference between committed offset and beginning_offset) and if the lag computed in the last n times is increasing you can scale up and vice versa. You might've to consider some edge cases for example in case consumers have gone down and lag would be increasing and the auto-scaling function might spawn more threads/machines).

这篇关于如何在消费者组kafka中动态添加消费者的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！