Kafka如何从__consumer_offsets主题阅读 [英] Kafka how to read from __consumer_offsets topic

问题描述

我正在尝试找出当前我的高级消费者正在解决的抵消措施.我使用Kafka 0.8.2.1，在Kafka的server.properties中设置了否"offset.storage"-我认为这意味着偏移量存储在Kafka中. (通过检查Zk shell中的以下路径，我还验证了Zookeeper中没有存储偏移量:/consumers/consumer_group_name/offsets/topic_name/partition_number)

I'm trying to find out which offsets my current High-Level consumers are working off. I use Kafka 0.8.2.1, with no "offset.storage" set in the server.properties of Kafka - which, I think, means that offsets are stored in Kafka. (I also verified that no offsets are stored in Zookeeper by checking this path in the Zk shell: /consumers/consumer_group_name/offsets/topic_name/partition_number )

我试图听听__consumer_offsets主题，以了解哪个使用者节省了偏移量的值，但是没有用...

I tried to listen to the __consumer_offsets topic to see which consumer saves what value of offsets, but it did not work...

我尝试了以下操作:

为控制台使用者创建了一个配置文件，如下所示:

created a config file for console consumer as following:

=> more kafka_offset_consumer.config 

 exclude.internal.topics=false

并尝试了两个版本的控制台使用者脚本:

and tried two versions of the console consumer scripts:

#1:
bin/kafka-console-consumer.sh --consumer.config kafka_offset_consumer.config --topic __consumer_offsets --zookeeper localhost:2181

#2
./bin/kafka-simple-consumer-shell.sh --topic __consumer_offsets --partition 0 --broker-list localhost:9092 --formatter "kafka.server.OffsetManager\$OffsetsMessageFormatter" --consumer.config kafka_offset_consumer.config

两者都不起作用-即使消费者正在积极地使用/保存偏移量，它也只能坐在那里而不打印任何内容.

Neither worked - it just sits there but does not print anything, even though the consumers are actively consuming/saving offsets.

我还缺少其他一些配置/属性吗?

Am I missing some other configuration/properties ?

谢谢！

Marina

推荐答案

在尝试同时使用 __ consumer_offsets 主题时，我遇到了这个问题. 我设法弄清楚了不同的Kafka版本，并认为我会分享发现的东西

I came across this question when trying to also consume from the __consumer_offsets topic. I managed to figure it out for different Kafka versions and thought I'd share what I'd found

对于Kafka 0.8.2.x

For Kafka 0.8.2.x

注意:这使用Zookeeper连接

Note: This uses Zookeeper connection

#Create consumer config
echo "exclude.internal.topics=false" > /tmp/consumer.config
#Consume all offsets
./kafka-console-consumer.sh --consumer.config /tmp/consumer.config \
--formatter "kafka.server.OffsetManager\$OffsetsMessageFormatter" \
--zookeeper localhost:2181 --topic __consumer_offsets --from-beginning

对于Kafka 0.9.x.x和0.10.x.x

For Kafka 0.9.x.x and 0.10.x.x

#Create consumer config
echo "exclude.internal.topics=false" > /tmp/consumer.config
#Consume all offsets
./kafka-console-consumer.sh --new-consumer --consumer.config /tmp/consumer.config \
--formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" \
--bootstrap-server localhost:9092 --topic __consumer_offsets --from-beginning

对于0.11.x.x-2.x

For 0.11.x.x - 2.x

#Create consumer config
echo "exclude.internal.topics=false" > /tmp/consumer.config
#Consume all offsets
./kafka-console-consumer.sh --consumer.config /tmp/consumer.config \
--formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" \
--bootstrap-server localhost:9092 --topic __consumer_offsets --from-beginning

这篇关于Kafka如何从__consumer_offsets主题阅读的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！