Kafka 节点突然从偏移量 0 开始消耗 [英] Kafka-node suddenly consumes from offset 0

问题描述

有时，kafka-node 消费者从偏移量 0 开始消费，而其默认行为仅消费较新的消息.然后它不会切换回其默认行为.你知道如何解决这个问题吗?会发生什么，它的行为会突然改变吗?代码非常简单，无需更改代码即可实现.

Sometimes, kafka-node consumer starts consuming from offset 0, while its default behavior it to consume only newer messages. Then it will not switch back to its default behavior. Do you know how to solve this and what happens and its behavior suddenly changes? The code is very simple and this happens without altering the code.

var kafka = require("kafka-node") ;
  Consumer = kafka.Consumer;
  client = new kafka.KafkaClient();


  consumer = new Consumer(client, [{ topic: "Topic_23", partition: 0}
                                    ]);


consumer.on("message", function(message) {

    console.log(message)


  });

到目前为止，我找到的唯一解决方案是更改 kafka 主题.然后一切正常.有任何想法吗 ?

The only solution I have found so far is to change the kafka topic. Then everything works fine again. Any ideas ?

推荐答案

在 Kafka 中，偏移量不与特定的消费者相关联，而是与消费者组相关联.在您的代码中，您没有提供消费者组，因此，每次启动消费者时，它都会被分配给不同的消费者组，因此偏移量从 0 开始.

In Kafka, offsets are not associated to specific consumers but instead, they are linked to the Consumer Groups. In your code, you don't provide the Consumer Group therefore, every time you fire up the consumer, it is being assigned to a different Consumer Group and thus, the offset starts from 0.

以下应该可以解决问题(显然是您第一次阅读所有消息时):

The following should do the trick (obviously the first time you are going to read all the messages):

var kafka = require("kafka-node") ;

Consumer = kafka.Consumer;
client = new kafka.KafkaClient();

payload = [{
    topic: "Topic_23", 
    partition: 0
}]

var options = {
    groupId: 'test-consumer-group',
    fromOffset: 'latest'
};


consumer = new Consumer(client, payload, options);
consumer.on("message", function(message) {
    console.log(message)
  });

这篇关于Kafka 节点突然从偏移量 0 开始消耗的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！