卡夫卡撞车后，偏移量消失了 [英] After kafka crashed, the offsets are lost

问题描述

我们的kafka系统崩溃，因为没有可用的磁盘空间. 使用者是使用Kafka Streams API的Spring启动应用程序. 现在，每个消费者应用程序都显示以下错误:

Our kafka system crashed because no disk space was available. The consumers are Spring boot application which are using the Kafka Streams API. Now every consumer application shows the following error:

java.io.FileNotFoundException: /tmp/kafka-streams/908a79bc-92e7-4f9c-a63a-5030cf4d3555/streams.device-identification-parser/0_48/.checkpoint.tmp (No such file or directory)

重新启动kafka服务器后，才发生此异常. 如果我们重新启动应用程序，该服务将从偏移量0开始，以重播消耗的主题中的所有消息.这是致命的.

This exception occurred exactly after the kafka server was restarted. If we restart the application, the service starts at offset 0 to replay all message from the consumed topic. This is fatal.

我们检查了日志:在每个分区上没有信息(示例行，我们有60个分区):

We checked the logs: On each partition there is no information (example line, we have 60 partitions):

TOPIC                               PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                            HOST            CLIENT-ID
device-identification.received      0          -               0               -               streams.device-identification-parser-3e81ef05-b9e3-495c-85b3-7e2912f00578-StreamThread-1-consumer-ca723291-3258-46c0-8b23-233bdc1e1191 /XX.X.X.X       streams.device-identification-parser-3e81ef05-b9e3-495c-85b3-7e2912f00578-StreamThread-1-consumer

有人可以帮助我恢复服务从最后一个偏移量开始的偏移量信息吗?

Can anyone help me, to restore the offset information, that the service starts from the last offset?

推荐答案

如果这是一个Kafka Streams应用程序，则您可能成功重置了它，可以使用Confluent的应用程序重置工具来完成:

If this is a Kafka Streams app you might be successful in resetting it, that can be done using Confluent's Application reset tool: https://docs.confluent.io/current/streams/developer-guide/app-reset-tool.html

这篇关于卡夫卡撞车后，偏移量消失了的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！