Kafka 如何从 __consumer_offsets 主题中读取 [英] Kafka how to read from __consumer_offsets topic
问题描述
我正在尝试找出我当前的高级消费者正在解决哪些抵消.我使用 Kafka 0.8.2.1,在 Kafka 的 server.properties 中设置nooffset.storage" - 我认为这意味着偏移量存储在 Kafka 中.(我还通过在 Zk shell 中检查此路径来验证 Zookeeper 中没有存储偏移量:/consumers/consumer_group_name/offsets/topic_name/partition_number
)
I'm trying to find out which offsets my current High-Level consumers are working off. I use Kafka 0.8.2.1, with no "offset.storage" set in the server.properties of Kafka - which, I think, means that offsets are stored in Kafka. (I also verified that no offsets are stored in Zookeeper by checking this path in the Zk shell: /consumers/consumer_group_name/offsets/topic_name/partition_number
)
我试着听__consumer_offsets
话题,看看哪个消费者保存了什么偏移值,但是没有用...
I tried to listen to the __consumer_offsets
topic to see which consumer saves what value of offsets, but it did not work...
我尝试了以下方法:
为控制台消费者创建了一个配置文件,如下所示:
created a config file for console consumer as following:
=> more kafka_offset_consumer.config
exclude.internal.topics=false
并尝试了两个版本的控制台使用者脚本:
and tried two versions of the console consumer scripts:
#1:
bin/kafka-console-consumer.sh --consumer.config kafka_offset_consumer.config --topic __consumer_offsets --zookeeper localhost:2181
#2
./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets --partition 0 --broker-list localhost:9092 --formatter "kafka.server.OffsetManager\$OffsetsMessageFormatter" --consumer.config kafka_offset_consumer.config
都不起作用 - 它只是坐在那里但不打印任何东西,即使消费者正在积极消费/保存抵消.
Neither worked - it just sits there but does not print anything, even though the consumers are actively consuming/saving offsets.
我是否缺少其他一些配置/属性?
Am I missing some other configuration/properties ?
谢谢!
玛丽娜
推荐答案
我在尝试从 __consumer_offsets 主题中消费时遇到了这个问题.我设法为不同的 Kafka 版本找出了答案,并认为我会分享我的发现
I came across this question when trying to also consume from the __consumer_offsets topic. I managed to figure it out for different Kafka versions and thought I'd share what I'd found
对于 Kafka 0.8.2.x
For Kafka 0.8.2.x
注意:这里使用 Zookeeper 连接
Note: This uses Zookeeper connection
#Create consumer config
echo "exclude.internal.topics=false" > /tmp/consumer.config
#Consume all offsets
./kafka-console-consumer.sh --consumer.config /tmp/consumer.config \
--formatter "kafka.server.OffsetManager\$OffsetsMessageFormatter" \
--zookeeper localhost:2181 --topic __consumer_offsets --from-beginning
对于 Kafka 0.9.x.x 和 0.10.x.x
For Kafka 0.9.x.x and 0.10.x.x
#Create consumer config
echo "exclude.internal.topics=false" > /tmp/consumer.config
#Consume all offsets
./kafka-console-consumer.sh --new-consumer --consumer.config /tmp/consumer.config \
--formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" \
--bootstrap-server localhost:9092 --topic __consumer_offsets --from-beginning
对于 0.11.x.x - 2.x
For 0.11.x.x - 2.x
#Create consumer config
echo "exclude.internal.topics=false" > /tmp/consumer.config
#Consume all offsets
./kafka-console-consumer.sh --consumer.config /tmp/consumer.config \
--formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" \
--bootstrap-server localhost:9092 --topic __consumer_offsets --from-beginning
这篇关于Kafka 如何从 __consumer_offsets 主题中读取的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!