如何重置Kafka偏移量以匹配尾巴位置? [英] How to reset Kafka offsets to match tail position?

问题描述

我们正在将Storm与Kafka和ZooKeeper一起使用.在这种情况下，我们必须删除一些主题，然后使用不同的名称重新创建它们.除了从新主题名称中读取内容外，我们的Kafka喷口保持不变.但是，现在，当尝试从新主题中读取时，喷口正在使用旧主题分区中的偏移量.因此，"my-topic-name"分区0的尾部位置将为500，但偏移量将为10000.

We're using Storm with Kafka and ZooKeeper. We had a situation where we had to delete some topics and recreate them with different names. Our Kafka spouts stayed the same, aside from now reading from the new topic names. However now the spouts are using the offsets from the old topic partitions when trying to read from the new topics. So the tail position of my-topic-name partition 0 will be 500 but the offset will be something like 10000.

有没有一种方法可以重置偏移位置，使其与主题的尾部匹配?

Is there a way to reset the offset position so it matches the tail of the topic?

推荐答案

有多个选项(因为Storm的KafkaSpout不提供任何用于定义起始偏移量的API).

There a multiple options (as Storm's KafkaSpout does not provide any API to define the starting offset).

1. 如果要从日志尾部使用，则应删除旧的偏移量
   • 取决于您的Kafka版本
     • (0.9之前的版本)您可以操纵ZK(有点棘手)
     • (0.9+)或尝试从主题__consumer_offsets中删除偏移量(这也很棘手，可能还会删除您要保留的其他偏移量)

1. If you want to consumer from the tail of the log you should delete old offsets
   • depending on you Kafka version
     • (pre 0.9) you can manipulate ZK (which is a little tricky)
     • (0.9+) or you try do delete the offset from the topic __consumer_offsets (which is also tricky and might delete other offset you want to preserve, too)

• 为此，您可以选择到日志末尾并提交
• 或者您提交了无效的偏移量(如-1)，并依赖最新"或最大"自动偏移量重置配置(取决于您的Kafka版本)

对于Kafka Streams，有一个应用程序重置工具"，可以执行类似的操作来处理已提交的偏移量.如果您想获取一些详细信息，可以阅读此博客文章 http://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/

For Kafka Streams, there is a "Application Reset Tool" that does a similar thing to manipulate committed offsets. If you want to get some details, you can read this blog post http://www.confluent.io/blog/data-reprocessing-with-kafka-streams-resetting-a-streams-application/

(免责声明:我是该帖子的作者，它是关于Kafka Streams的-但是，基本的偏移量操作思想是相同的)

(disclaimer: I am the author of the post and it is about Kafka Streams -- nevertheless, the underlying offset manipulation ideas are the same)

这篇关于如何重置Kafka偏移量以匹配尾巴位置?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！