基于优先级或循环方式从两个flink dataStream 中消费 [英] Consume from two flink dataStream based on priority or round robin way

问题描述

我有两个 flink dataStream.例如:dataStream1 和 dataStream2.我想将两个流合并为 1 个流，以便我可以使用相同的处理函数来处理它们，因为 dataStream 的 dag 是相同的.

I have two flink dataStream. For ex: dataStream1 and dataStream2. I want to union both the Streams into 1 stream so that I can process them using the same process functions as the dag of both dataStream is the same.

截至目前，我需要对任一流的消息消费具有相同的优先级.dataStream2 的生产者每分钟产生 10 条消息，而 dataStream1 的生产者每秒产生 1000 条消息.此外，两个 dataStreams.DataSteam2 的数据类型是相同的，更多的是应该尽快使用的高优先级队列.dataStream1 和 dataStream2 的消息之间没有关系

As of now, I need equal priority of consumption of messages for either stream. The producer of dataStream2 produces 10 messages per minute, while the producer of dataStream1 produces 1000 messages per second. Also, dataTypes are the same for both dataStreams.DataSteam2 more of a high priority queue that should be consumed asap. There is no relation between messages of dataStream1 and dataStream2

dataStream1.union(dataStream2) 会产生一个包含两个流的元素的流吗?

Does dataStream1.union(dataStream2) will produce a Stream that will have elements of both Streams?

推荐答案

可能是这个问题最简单的解决方案，但并不是最有效的解决方案，具体取决于您的数据源的确切规范，可能是连接两个流.在此解决方案中，您可以使用 CoProcessFunction，它将为每个连接的流调用单独的方法.

Probably the simplest solution to this problem, yet not exactly the most efficient one depending on the exact specification of the sources for Your data, may be connecting the two streams. In this solution, You could use the CoProcessFunction, which will invoke separate methods for each of the connected streams.

在此解决方案中，您可以简单地缓冲一个流的元素，直到可以生成它们(例如以循环方式).但请记住，如果源产生事件的频率之间存在很大差异，这可能会非常低效.

In this solution, You could simply buffer the elements of one stream until they can be produced (for example in round-robin manner). But keep in mind that this may be quite inefficient if there is a very big difference between the frequency in which sources produce events.

这篇关于基于优先级或循环方式从两个flink dataStream 中消费的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！