如何从kafka服务器获取主题中的所有消息 [英] how to get the all messages in a topic from kafka server

问题描述

我想从服务器中的某个主题开始获取所有消息.

I would like to get all the messages from beginning in a topic from server.

例如:

bin/kafka-console-consumer.sh --zookeeper本地主机:2181 --topic testTopic --from-beginning

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic testTopic --from-beginning

使用上述控制台命令时，我希望能够从一开始就获取主题中的所有消息，但是我无法从一开始就使用Java代码来消耗主题中的所有消息.

When using the above console command, I would like to able to get all messages in a topic from the beginning but I couldn't consume all the messages in a topic from beginning using java code.

推荐答案

最简单的方法是启动使用者并耗尽所有消息.现在，我不知道您的主题中有多少个分区以及您是否已经有一个现有的消费者组，但是您有几个选择:

The easiest way would be to start a consumer and drain all the messages. Now I don't know how many partitions you have in your topic and whether you already have a an existing consumer group or not, but you have a few options:

看看这个API: 1)如果您在同一个使用者组中已经有一个使用者，并且仍然想从头开始使用，则应使用API​​文档中列出的seek选项，并将每个使用者的偏移量设置为0.群组.这将从一开始就开始消耗.

1) If you already have a consumer in the same consumer group, and still want to start consuming from the beginning, you should use the seek option listed in the API doc and set the offset to 0 for each consumer in the group. This would start consuming from the beginning.

2)否则，您可以在新的消费群体中启动一些消费群体.您不必担心寻找.

2) Otherwise, you can start a few consumers in a new consumer group & you would not have to worry about seek.

PS:如果您对Kafka有更多疑问，请记住以后提供有关您的设置的更多详细信息.很多事情取决于您如何配置基础架构和应用程序.您希望它的状态如何，因此会因情况而异.

PS: Please remember to provide more details about your setup in the future if you have more questions on Kafka. A lot of things depend on how you have configured your infrastructure & how you would prefer it to be and would thus vary from case to case.

这篇关于如何从kafka服务器获取主题中的所有消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！