kafka-python 消费者未收到消息 [英] kafka-python consumer not receiving messages

问题描述

我在使用 KafaConsumer 时遇到问题，无法从头读取，或从任何其他显式偏移量读取.

I am having trouble with KafaConsumer to make it read from the beginning, or from any other explicit offset.

为同一主题的消费者运行命令行工具，我确实看到带有 --from-beginning 选项的消息，否则它会挂起

Running the command line tools for the consumer for the same topic , I do see messages with the --from-beginning option and it hangs otherwise

$ ./kafka-console-consumer.sh --zookeeper {localhost:port} --topic {topic_name} --from-beginning

如果我通过 python 运行它，它会挂起，我怀疑这是由不正确的消费者配置引起的

If I run it through python, it hangs, which I suspect to be caused by incorrect consumer configs

consumer = KafkaConsumer(topic_name,
                     bootstrap_servers=['localhost:9092'],
                     group_id=None,
                     auto_commit_enable=False,
                     auto_offset_reset='smallest')

print "Consuming messages from the given topic"
for message in consumer:
    print "Message", message
    if message is not None:
        print message.offset, message.value

print "Quit"

输出:

使用给定主题的消息(在那之后挂起)

Output:

Consuming messages from the given topic (hangs after that)

我使用 kafka-python 0.9.5，代理运行 kafka 8.2.不确定确切的问题是什么.

I am using kafka-python 0.9.5 and the broker runs kafka 8.2. Not sure what the exact problem is.

按照 dpkp 的建议设置 _group_id=None_ 以模拟控制台消费者的行为.

Set _group_id=None_ as suggested by dpkp to emulate the behavior of console consumer.

推荐答案

console-consumer 和你贴的python consumer 代码的区别是python consumer 使用consumer group 来保存偏移量:group_id="测试消费者组" .如果您改为设置 group_id=None，您应该看到与控制台使用者相同的行为.

The difference between the console-consumer and the python consumer code you have posted is the python consumer uses a consumer group to save offsets: group_id="test-consumer-group" . If instead you set group_id=None, you should see the same behavior as the console consumer.

这篇关于kafka-python 消费者未收到消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！