Apache Flink DataStream API没有mapPartition转换 [英] Apache Flink DataStream API doesn't have a mapPartition transformation

问题描述

Spark DStream具有mapPartition API，而Flink DataStream API没有.有没有人可以帮助解释原因.我想做的是在Flink上实现类似于Spark reduceByKey的API.

Spark DStream has mapPartition API, while Flink DataStream API doesn't. Is there anyone who could help explain the reason. What I want to do is to implement a API similar to Spark reduceByKey on Flink.

推荐答案

Flink的流处理模型与Spark Streaming完全不同，后者以小批处理为中心.在Spark Streaming中，每个小型批处理都像常规批处理程序一样在有限的数据集上执行，而Flink DataStream程序则连续处理记录.

Flink's stream processing model is quite different from Spark Streaming which is centered around mini batches. In Spark Streaming each mini batch is executed like a regular batch program on a finite set of data, whereas Flink DataStream programs continuously process records.

在Flink的DataSet API中，MapPartitionFunction具有两个参数.输入的迭代器和函数的结果的收集器. Flink DataStream程序中的MapPartitionFunction永远不会从第一个函数调用中返回，因为迭代器将迭代无休止的记录流.但是，Flink的内部流处理模型要求用户功能返回以便检查点功能状态.因此，DataStream API不提供mapPartition转换.

In Flink's DataSet API, a MapPartitionFunction has two parameters. An iterator for the input and a collector for the result of the function. A MapPartitionFunction in a Flink DataStream program would never return from the first function call, because the iterator would iterate over an endless stream of records. However, Flink's internal stream processing model requires that user functions return in order to checkpoint function state. Therefore, the DataStream API does not offer a mapPartition transformation.

为了实现类似于Spark Streaming的reduceByKey的功能，您需要在流上定义一个键控窗口. Windows将流离散化，这有点类似于小型批处理，但是Windows提供了更大的灵活性.由于窗口的大小是有限的，因此可以调用reduce窗口.

In order to implement functionality similar to Spark Streaming's reduceByKey, you need to define a keyed window over the stream. Windows discretize streams which is somewhat similar to mini batches but windows offer way more flexibility. Since a window is of finite size, you can call reduce the window.

这看起来像:

yourStream.keyBy("myKey") // organize stream by key "myKey"
          .timeWindow(Time.seconds(5)) // build 5 sec tumbling windows
          .reduce(new YourReduceFunction); // apply a reduce function on each window

DataStream文档显示了如何定义各种窗口类型并说明所有可用功能.

The DataStream documentation shows how to define various window types and explains all available functions.

注意:最近重新设计了DataStream API.该示例假定使用最新版本(0.10-SNAPSHOT)，并将在未来几天以0.10.0发行.

Note: The DataStream API has been reworked recently. The example assumes the latest version (0.10-SNAPSHOT) which will be release as 0.10.0 in the next days.

这篇关于Apache Flink DataStream API没有mapPartition转换的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！