Kafka 0.9新的消费者API ---如何只看消费者抵销 [英] Kafka 0.9 new consumer api --- how to just watch consumer offsets

问题描述

我正在尝试使用Java API监视给定组的使用者偏移量.我创建了一个额外的使用者，该使用者不订阅任何主题，而只是调用consumer.committed(topic)来获取偏移量信息.这种作品，但是:

I am trying to monitor consumer offsets of a given group with the Java API. I create one additional consumer which does not subscribe to any topic, but just calls consumer.committed(topic) to get the offset information. This kind of works, but:

为了进行测试，我仅使用一个真实的消费者(即确实订阅该主题的消费者).当我使用close()关闭它并随后重新启动它时，尽管我使用了poll(1000)，但从订阅到第一次使用消息之间要花费27秒.

For testing I use only one real consumer (i.e. one which does subscribe to the topic). When I shut it down using close() and later restart one, it takes 27 seconds between subscribe and the first consumption of messages, despite the fact that I use poll(1000).

我猜想这与非订阅用户可能会混淆的重新平衡有关.那有可能吗?是否有更好的方法来使用Java API监视偏移量(我了解命令行工具，但需要使用API​​).

I am guessing this has to do with the rebalancing possibly being confused by the non-subscribing consumer. Could that be possible? Is there a better way to monitor offsets with the Java API (I know about the command line tool, but need to use the API).

推荐答案

找到了它:监控用户加剧了混乱，但不是罪魁祸首.最后，尽管有些意外(至少对我来说)，还是很容易理解的:

Found it: the monitoring consumer added to the confusion but was not the culprit. In the end it is easy to understand though slightly unexpected (for me at least):

session.timeout.ms的默认值为30秒.当消费者消失时，最多需要30秒才能宣布消费者死亡并重新平衡工作.为了进行测试，我停止了所拥有的单个消费者，等待了三秒钟，然后重新启动了一个新的消费者.然后花了27秒才开始，填补了30秒的超时时间.

The default for session.timeout.ms is 30 seconds. When a consumer disappears it takes up to 30 seconds before it is declared dead and the work is rebalanced. For testing, I had stopped the single consumer I had, waited three seconds and restarted a new one. This then took 27 seconds before it started, filling the 30 seconds time-out.

我本来希望一个单独的消费者启动不会等待超时到期，而是会开始重新平衡"，即立即抓紧工作.看来，即使只有一个消费者，超时也必须在重新平衡工作之前到期.

I would have expected that a single, lone consumer starting up does not wait for the time-out to expire, but starts to "rebalance", i.e. grab the work immediately. It seems though that the time-out has to expire before work is rebalanced, even if there is only one consumer.

为了使测试更快通过，我更改了配置，以将较低的session.timeout.ms用于消费者，将group.min.session.timeout.ms用于代理.

For the testing to get through faster, I changed the configuration to use a lower session.timeout.ms for the consumer as well as group.min.session.timeout.ms for the broker.

得出结论:使用不订阅任何主题来监视偏移量的使用者效果很好，并且似乎不会干扰重新平衡过程.

To conclude: using a consumer that does not subscribe to any topic for monitoring the offsets works just fine and does not seem to interfere with the rebalancing process.

这篇关于Kafka 0.9新的消费者API ---如何只看消费者抵销的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！