Kafka节点突然从偏移量0消耗 [英] Kafka-node suddenly consumes from offset 0

问题描述

有时，kafka-node使用者从偏移量0开始消耗，而其默认行为是仅消耗较新的消息.然后，它将不会切换回其默认行为.您知道如何解决这个问题，会发生什么并且它的行为突然改变吗?代码非常简单，并且无需更改代码即可实现.

Sometimes, kafka-node consumer starts consuming from offset 0, while its default behavior it to consume only newer messages. Then it will not switch back to its default behavior. Do you know how to solve this and what happens and its behavior suddenly changes? The code is very simple and this happens without altering the code.

var kafka = require("kafka-node") ;
  Consumer = kafka.Consumer;
  client = new kafka.KafkaClient();


  consumer = new Consumer(client, [{ topic: "Topic_23", partition: 0}
                                    ]);


consumer.on("message", function(message) {

    console.log(message)


  });

到目前为止，我发现的唯一解决方案是更改kafka主题.然后，一切再次正常.有任何想法吗 ?

The only solution I have found so far is to change the kafka topic. Then everything works fine again. Any ideas ?

推荐答案

在Kafka中，偏移量不与特定使用者相关联，而是与消费者组相关联.在您的代码中，您没有提供使用者组，因此，每次启动使用者时，它都会被分配给另一个使用者组，因此，偏移量从0开始.

In Kafka, offsets are not associated to specific consumers but instead, they are linked to the Consumer Groups. In your code, you don't provide the Consumer Group therefore, every time you fire up the consumer, it is being assigned to a different Consumer Group and thus, the offset starts from 0.

以下应该可以解决问题(很明显，这是您第一次阅读所有消息时):

The following should do the trick (obviously the first time you are going to read all the messages):

var kafka = require("kafka-node") ;

Consumer = kafka.Consumer;
client = new kafka.KafkaClient();

payload = [{
    topic: "Topic_23", 
    partition: 0
}]

var options = {
    groupId: 'test-consumer-group',
    fromOffset: 'latest'
};


consumer = new Consumer(client, payload, options);
consumer.on("message", function(message) {
    console.log(message)
  });

这篇关于Kafka节点突然从偏移量0消耗的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！