kafka 崩溃后，偏移量丢失 [英] After kafka crashed, the offsets are lost
问题描述
我们的 kafka 系统崩溃了，因为没有可用的磁盘空间.消费者是使用 Kafka Streams API 的 Spring 启动应用程序.现在每个消费者应用程序都显示以下错误:
Our kafka system crashed because no disk space was available. The consumers are Spring boot application which are using the Kafka Streams API. Now every consumer application shows the following error:
java.io.FileNotFoundException: /tmp/kafka-streams/908a79bc-92e7-4f9c-a63a-5030cf4d3555/streams.device-identification-parser/0_48/.checkpoint.tmp (No such file or directory)
这个异常恰好在kafka服务器重启后发生.如果我们重新启动应用程序，该服务将从偏移量 0 开始重放来自已消费主题的所有消息.这是致命的.
This exception occurred exactly after the kafka server was restarted. If we restart the application, the service starts at offset 0 to replay all message from the consumed topic. This is fatal.
我们检查了日志:在每个分区上都没有信息(例如，我们有 60 个分区):
We checked the logs: On each partition there is no information (example line, we have 60 partitions):
TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
device-identification.received 0 - 0 - streams.device-identification-parser-3e81ef05-b9e3-495c-85b3-7e2912f00578-StreamThread-1-consumer-ca723291-3258-46c0-8b23-233bdc1e1191 /XX.X.X.X streams.device-identification-parser-3e81ef05-b9e3-495c-85b3-7e2912f00578-StreamThread-1-consumer
谁能帮我恢复偏移量信息，服务从最后一个偏移量开始?
Can anyone help me, to restore the offset information, that the service starts from the last offset?
推荐答案
如果这是一个 Kafka Streams 应用程序，您可能会成功重置它，这可以使用 Confluent 的应用程序重置工具完成:https://docs.confluent.io/current/streams/developer-guide/app-重置工具.html
If this is a Kafka Streams app you might be successful in resetting it, that can be done using Confluent's Application reset tool: https://docs.confluent.io/current/streams/developer-guide/app-reset-tool.html
这篇关于kafka 崩溃后，偏移量丢失的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
