关于 Flink 一次特性 [英] About Flink exactly-once feature

问题描述

我正在阅读有关 flink 一次功能的文档 此处.而且有些句子我不太明白:

I am reading the documentation about flink exactly-once feature here. And I do not quite understand some of the sentences:

成功预提交后，必须保证提交最终成功——我们的运营商和我们的外部系统都需要做出这一保证.如果一次提交失败(例如，由于间歇性网络问题)，整个 Flink 应用程序失败，根据用户的重启策略重新启动，并有另一次提交尝试.这个过程很关键，因为如果提交最终没有成功，就会发生数据丢失.

After a successful pre-commit, the commit must be guaranteed to eventually succeed – both our operators and our external system need to make this guarantee. If a commit fails (for example, due to an intermittent network issue), the entire Flink application fails, restarts according to the user’s restart strategy, and there is another commit attempt. This process is critical because if the commit does not eventually succeed, data loss occurs.

这表示如果提交最终没有成功，就会发生数据丢失.我将其解释为:提交可能会成功，但由于某些原因，每次重启都会失败.在这种情况下，Flink 只能放弃属于这个 commit 的数据.那么，如果数据丢失是不可接受的，应用程序应该重新启动直到提交成功?

This says data loss occurs if the commit does not eventually succeed. I interpret it as: The commit could succeed but it just happen to keep failing for every restart because of certain reason. In this case, Flink can only give up on the data belonging to this commit. So, if data loss is unacceptable, the application should be restarted until the commit succeeds ?

我们知道，如果出现任何故障，Flink 会将应用程序的状态恢复到最新的成功检查点.一个潜在的问题是在一种罕见的情况下，失败发生在成功的预提交之后但在该事实(提交)通知到达我们的操作员之前.在这种情况下，Flink 会将我们的算子恢复到已经预提交但尚未提交的状态.

As we know, if there’s any failure, Flink restores the state of the application to the latest successful checkpoint. One potential catch is in a rare case when the failure occurs after a successful pre-commit but before notification of that fact (a commit) reaches our operator. In that case, Flink restores our operator to the state that has already been pre-committed but not yet committed.

我也不太关注这里.这个通知是关于什么的，上面没有提到?所说的运营商是指汇运营商吗?另外，按照我的理解，如果commit成功了，只有所谓的notification失败，那么恢复到pre-commited状态后会不会造成数据重复?

I do not quite follow here, either. What's this notification about, which is not mentioned above ? And does the said operator mean the sink operator ? Also, as I interpret it, if the commit has succeeded and only the so-called notification fails, would it cause data duplication after restoration to the pre-commited state ?

如果问题本身无效，请纠正我.任何帮助表示赞赏.

Please correct me if the question itself is not valid. Any help is appreciated.

推荐答案

Flink 的端到端恰好一次机制基于类似两阶段提交 (2PC) 的协议.该协议用于协调程序的一个或所有接收器将输出提交给外部系统.

Flink's end-to-end exactly-once mechanism is based on a two phase commit (2PC) like protocol. The protocol is used to coordinate that either none or all sinks of a program commit output to an external system.

当接收器任务说我准备提交"(预提交)时，它保证它能够执行提交.接收器任务然后等待接收来自协调器的提交通知，只有当所有接收器任务都同意准备提交时才会发送该通知.如果在收到通知之前申请失败，则该保证也必须成立.在这种情况下，接收器任务必须能够恢复打开(尚未提交)的事务并在收到下一个通知时执行它.在多次失败的情况下，接收器必须继续尝试直到提交成功.但是，即使在一个(或多个)失败的情况下，事务也只能执行一次.

When a sink task says "I am ready to commit" (pre-commit), it gives the guarantee that it is able to perform the commit. The sink task then waits to receive a commit notification from the coordinator which is only sent if all sink tasks agreed on being ready to commit. The guarantee must also hold if the application fails before the notification is received. In that case, the sink task must be able to recover the open (not-yet-committed) transaction and execute it when the next notification is received. In case of multiple failures, the sink must keep on trying until the commit succeeds. However, the transaction must only be performed once, even in case of one (or more) failure.

就是这个意思

预提交成功后，必须保证提交最终成功

After a successful pre-commit, the commit must be guaranteed to eventually succeed

如果接收器任务最终无法提交它预先提交的数据，则数据将丢失.

If a sink task is not able to eventually commit data that it pre-committed, the data is lost.

这篇关于关于 Flink 一次特性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！