Flink Kinesis使用者未存储最后成功处理的序列号 [英] Flink Kinesis Consumer not storing last successfully processed sequence nos

问题描述

我们正在使用Flink Kinesis Consumer将来自Kinesis流的数据消费到我们的Flink应用程序中.

We are using Flink Kinesis Consumer to consume data from Kinesis stream into our Flink application.

KCL库使用DynamoDB表存储最后成功处理的Kinesis流序列号.以便下次启动应用程序时，它将从上次停止的地方恢复.

KCL library uses a DynamoDB table to store last successfully processed Kinesis stream sequence nos. so that the next time application starts, it resumes from where it left off.

但是，似乎Flink Kinesis Consumer没有维护任何此类序列号.在任何持久性存储中.因此，我们需要依靠ShardIteratortype(trim_horizen，最新等)来决定在应用程序重新启动时在哪里恢复Flink应用程序处理.

But, it seems that Flink Kinesis Consumer does not maintain any such sequence nos. in any persistent store. As a result, we need to rely upon ShardIteratortype (trim_horizen, latest, etc) to decide where to resume Flink application processing upon application restart.

对此的一种可能的解决方案是依靠Flink检查点机制，但是仅当应用程序在失败时恢复时才起作用，而不是在应用程序被有意取消并需要从最近成功使用的Kinesis流序列中重新启动时才起作用不.

A possible solution to this could be to rely on Flink checkpointing mechanism, but that only works when application resumes upon failure, and not when the application has been deliberately cancelled and is needed to be restarted from the last successfully consumed Kinesis stream sequence no.

我们是否需要自己存储这些最后一次成功使用的序列?

Do we need to store these last successfully consumed sequence nos ourselves ?

推荐答案

Flink的最佳实践是使用检查点和保存点，因为它们创建一致的快照，这些快照包含到消息队列中的偏移量(在这种情况下，为Kinesis流序列号)以及整个工作图的所有状态，这些状态是由于消耗了数据直到这些偏移而导致的.这样就可以恢复或重新启动，而不会丢失或重复数据.

Best practice with Flink is to use checkpoints and savepoints, as these create consistent snapshots that contain offsets into your message queues (in this case, Kinesis stream sequence numbers) together with all of the state throughout the rest of the job graph that resulted from having consumed the data up to those offsets. This makes it possible to recover or restart without any loss or duplication of data.

Flink的检查点是Flink自身自动拍摄的快照，目的是从故障中恢复，并且采用可快速恢复的优化格式. 保存点使用相同的底层快照机制，但是是手动触发的，其格式更关注操作的灵活性而不是性能.

Flink's checkpoints are snapshots taken automatically by Flink itself for the purpose of recovery from failures, and are in a format optimized for rapid restoration. Savepoints use the same underlying snapshot mechanism, but are triggered manually, and their format is more concerned about operational flexibility than performance.

保存点是您要寻找的.特别是从保存点恢复非常有用.

Savepoints are what you are looking for. In particular, cancel with savepoint and resume from savepoint are very useful.

另一种选择是使用 查看全文