详细信息:Flink如何实现一次精确机制? [英] Details: How Flink achieves exactly-once mechanism?

问题描述

从以前的帖子看来，Flink可以一次实现完全匹配

From a previous post, it seems Flink achieves exactly-once by

成功进行预提交后，必须保证提交最终成功

After a successful pre-commit, the commit must be guaranteed to eventually succeed

我认为Flink Task Manager实现了成功的预提交"；Flink接收器实现了最终成功".

I think "a successful pre-commit" is achieved by Flink Task Manager; and the "eventual succeed" is achieved by the Flink sink.

1. Flink接收器节点如何实现最终成功"?
2. 此一次机制与检查点是否有关系?

推荐答案

Flink的两阶段提交接收器通常通过以下方式将其操作与检查点机制耦合:

Flink's two-phase commit sinks typically couple their actions with the checkpointing mechanism in the following way:

• onSnapshot :刷新所有记录并预先提交
• onCheckpointComplete :提交待处理的事务并发布数据
• onRecovery :检查并提交所有未决事务

• onSnapshot: Flush all records and pre-commit
• onCheckpointComplete: Commit pending transactions and publish data
• onRecovery: Check and commit any pending transactions

请注意，如果外部系统超时在 onRecovery 阶段提交的未决事务，数据可能会丢失.

Note that it is possible for data to be lost if the external system times out pending transactions that would be committed during the onRecovery phase.

您可以在 Apache Flink中的端到端精确一次处理概述(也可以使用Apache Kafka！).

这篇关于详细信息:Flink如何实现一次精确机制?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！