如何重置 Kafka 偏移量以匹配尾部位置? [英] How to reset Kafka offsets to match tail position?

问题描述

我们将 Storm 与 Kafka 和 ZooKeeper 一起使用.我们遇到过必须删除一些主题并使用不同名称重新创建它们的情况.除了现在读取新主题名称之外，我们的 Kafka 喷口保持不变.但是现在，当尝试从新主题读取时，spout 正在使用旧主题分区的偏移量.因此，my-topic-name 分区 0 的尾部位置将为 500，但偏移量将类似于 10000.

We're using Storm with Kafka and ZooKeeper. We had a situation where we had to delete some topics and recreate them with different names. Our Kafka spouts stayed the same, aside from now reading from the new topic names. However now the spouts are using the offsets from the old topic partitions when trying to read from the new topics. So the tail position of my-topic-name partition 0 will be 500 but the offset will be something like 10000.

有没有办法重置偏移位置，使其与主题的尾部匹配?

Is there a way to reset the offset position so it matches the tail of the topic?

推荐答案

有多种选择(因为 Storm 的 KafkaSpout 没有提供任何 API 来定义起始偏移量).

There a multiple options (as Storm's KafkaSpout does not provide any API to define the starting offset).

1. 如果你想从日志的尾部消费你应该删除旧的偏移量
   • 取决于您的 Kafka 版本
     • (0.9 之前)你可以操纵 ZK(这有点棘手)
     • (0.9+) 或者您尝试从主题 __consumer_offsets 中删除偏移量(这也很棘手，可能还会删除您想要保留的其他偏移量)

1. If you want to consumer from the tail of the log you should delete old offsets
   • depending on you Kafka version
     • (pre 0.9) you can manipulate ZK (which is a little tricky)
     • (0.9+) or you try do delete the offset from the topic __consumer_offsets (which is also tricky and might delete other offset you want to preserve, too)

• 为此，您要么查找日志的末尾并提交
• 或者您提交了无效的偏移量(如 -1)并依赖自动偏移量重置配置最新"或最大"(取决于您的 Kafka 版本)

对于 Kafka Streams，有一个应用程序重置工具"可以做类似的事情来操作提交的偏移量.如果你想了解一些细节，你可以阅读这篇博文 http://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/

For Kafka Streams, there is a "Application Reset Tool" that does a similar thing to manipulate committed offsets. If you want to get some details, you can read this blog post http://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/

(免责声明:我是这篇文章的作者，它是关于 Kafka Streams 的——不过，底层的偏移操作思想是相同的)

(disclaimer: I am the author of the post and it is about Kafka Streams -- nevertheless, the underlying offset manipulation ideas are the same)

这篇关于如何重置 Kafka 偏移量以匹配尾部位置?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！