Flink Kinesis Consumer 不存储最后成功处理的序列号 [英] Flink Kinesis Consumer not storing last successfully processed sequence nos

问题描述

我们正在使用 Flink Kinesis Consumer 将 Kinesis 流中的数据消费到我们的 Flink 应用程序中.

We are using Flink Kinesis Consumer to consume data from Kinesis stream into our Flink application.

KCL 库使用 DynamoDB 表来存储最后成功处理的 Kinesis 流序列号.以便下次应用程序启动时，它会从停止的地方继续.

KCL library uses a DynamoDB table to store last successfully processed Kinesis stream sequence nos. so that the next time application starts, it resumes from where it left off.

但是，Flink Kinesis Consumer 似乎没有维护任何这样的序列号.在任何持久存储中.因此，我们需要依赖 ShardIteratortype(trim_horizen、latest 等)来决定在应用重启后从哪里恢复 Flink 应用处理.

But, it seems that Flink Kinesis Consumer does not maintain any such sequence nos. in any persistent store. As a result, we need to rely upon ShardIteratortype (trim_horizen, latest, etc) to decide where to resume Flink application processing upon application restart.

一个可能的解决方案是依赖 Flink 检查点机制，但只有当应用程序在失败时恢复时才有效，而不是当应用程序被故意取消并且需要从上次成功使用的 Kinesis 流序列重新启动时没有.

A possible solution to this could be to rely on Flink checkpointing mechanism, but that only works when application resumes upon failure, and not when the application has been deliberately cancelled and is needed to be restarted from the last successfully consumed Kinesis stream sequence no.

我们需要自己存储这些最后成功消费的序列号吗?

Do we need to store these last successfully consumed sequence nos ourselves ?

推荐答案

Flink 的最佳实践是使用检查点和保存点，因为它们会创建一致的快照，其中包含消息队列中的偏移量(在本例中为 Kinesis 流序列号)连同整个作业图其余部分的所有状态，这些状态是由于消耗了这些偏移量的数据而产生的.这使得在不丢失或重复数据的情况下恢复或重新启动成为可能.

Best practice with Flink is to use checkpoints and savepoints, as these create consistent snapshots that contain offsets into your message queues (in this case, Kinesis stream sequence numbers) together with all of the state throughout the rest of the job graph that resulted from having consumed the data up to those offsets. This makes it possible to recover or restart without any loss or duplication of data.

Flink 的检查点 是 Fl​​ink 自身自动拍摄的用于故障恢复的快照，并且采用了为快速恢复而优化的格式.Savepoints 使用相同底层快照机制，但都是手动触发的，其格式更关注操作灵活性而不是性能.

Flink's checkpoints are snapshots taken automatically by Flink itself for the purpose of recovery from failures, and are in a format optimized for rapid restoration. Savepoints use the same underlying snapshot mechanism, but are triggered manually, and their format is more concerned about operational flexibility than performance.

保存点就是您要找的.特别是，使用保存点取消 和 从保存点恢复非常有用.

Savepoints are what you are looking for. In particular, cancel with savepoint and resume from savepoint are very useful.

另一种选择是使用 保留检查点 与 ExternalizedCheckpointCleanup.RETAIN_ON_CANCELATION.

Another option is to use retained checkpoints with ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION.

这篇关于Flink Kinesis Consumer 不存储最后成功处理的序列号的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！