我的 Flink 应用程序需要水印吗?如果没有，我需要 WatermarkStrategy.noWatermarks 吗? [英] Does my Flink application need watermarks? If not, do I need WatermarkStrategy.noWatermarks?

问题描述

我不确定我的 Flink 应用程序是否真的需要水印.什么时候需要?

I'm not sure if my Flink application actually requires Watermarks. When are they necessary?

如果我不需要它们，WatermarkStrategy.noWatermarks() 的目的是什么?

And if I don't need them, what is the purpose of WatermarkStrategy.noWatermarks()?

推荐答案

时间 t 的水印标记数据流中的一个位置并断言该流现在已经完成通过时间t.

A Watermark for time t marks a location in a data stream and asserts that the stream, at that point, is now complete up through time t.

水印的唯一目的是触发基于事件时间的计时器.

The only purpose watermarks serve is to trigger the firing of event-time-based timers.

基于事件时间的计时器由 KeyedProcessFunction API 直接公开，也由

Event-time-based timers are directly exposed by the KeyedProcessFunction API, and are also used internally by

• 事件时间窗口
• CEP(模式匹配)库，如果您指定要进行基于事件时间的处理，它会使用水印对传入流进行排序
• Flink SQL，同样仅在进行基于事件时间的处理时:例如 ORDER BY、版本化表连接、windows、MATCH_RECOGNIZE 等.

不需要水印的常见情况包括仅依赖处理时间或进行批处理时的应用程序.或者在处理具有时间戳但从不依赖事件时间计时器的数据时(例如，简单的逐事件处理).

Common cases where you don't need watermarks include applications that only rely on processing time, or when doing batch processing. Or when processing data that has timestamps, but never relying on event-time timers (e.g., simple event-by-event processing).

由 FLIP-27 引入的 Flink 的新源接口确实需要 WatermarkStrategy:

Flink's new source interface, introduced by FLIP-27, does require a WatermarkStrategy:

env.fromSource(source, watermarkStrategy, sourceName);

如果你实际上不需要水印，你可以在这个接口中使用WatermarkStrategy.noWatermarks().

In cases where you don't actually need watermarks, you can use WatermarkStrategy.noWatermarks() in this interface.

这篇关于我的 Flink 应用程序需要水印吗?如果没有，我需要 WatermarkStrategy.noWatermarks 吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！