Apache Beam 如何管理运动检查点? [英] How Apache Beam manage kinesis checkpointing?

问题描述

我有一个在 Apache Beam(使用 Spark Runner)中开发的流式管道，它从 kinesis 流中读取数据.

I have a streaming pipeline developed in Apache Beam (using Spark Runner) which reads from kinesis stream.

我正在寻找 Apache Beam 中的选项来管理 kinesis 检查点(即定期存储 kinesis 流的当前位置)，以便它允许系统从故障中恢复并在流停止的地方继续处理.

I am looking out for options in Apache Beam to manage kinesis checkpointing (i.e. stores periodically the current position of kinesis stream) so as it allows the system to recover from failures and continue processing where the stream left off.

Apache Beam 是否提供了类似于 Spark Streaming 的支持 kinesis 检查点的规定(参考链接 - https://spark.apache.org/docs/2.2.0/streaming-kinesis-integration.html)?

Is there a provision available for Apache Beam to support kinesis checkpointing as similar to Spark Streaming (Reference link - https://spark.apache.org/docs/2.2.0/streaming-kinesis-integration.html)?

推荐答案

由于 KinesisIO 基于 UnboundedSource.CheckpointMark，它使用了 Beam 提供的标准检查点机制UnboundedSource.UnboundedReader.

Since KinesisIO is based on UnboundedSource.CheckpointMark, it uses the standard checkpoint mechanism, provided by Beam UnboundedSource.UnboundedReader.

一旦读取了 KinesisRecord(实际上，是通过从 Kinesis 分片中实际获取记录而从单独馈送的记录队列中提取的)，则分片检查点将是 更新 使用记录 SequenceNumber 然后，根据 UnboundedSource 的运行器实现和检查点处理，将被保存.

Once a KinesisRecord has been read (actually, pulled from a records queue that is feed separately by actually fetching the records from Kinesis shard), then the shard checkpoint will be updated by using the record SequenceNumber and then, depending on runner implementation of UnboundedSource and checkpoints processing, will be saved.

Afaik，Beam Spark Runner 为此目的使用 Spark 状态机制.

Afaik, Beam Spark Runner uses Spark States mechanism for this purposes.

这篇关于Apache Beam 如何管理运动检查点?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！