如何根据数据将一个数据流输出到不同的输出? [英] How to output one data stream to different outputs depending on the data?
问题描述
在Apache Flink中,我有一个元组流.让我们假设一个非常简单的Tuple1<String>
.元组的值字段中可以有任意值(例如'P1','P2'等).可能的值的集合是有限的,但我事先不知道完整的集合(因此可能会有一个"P362").我想根据元组内部的值将该元组写入某个输出位置.所以我希望具有以下文件结构:
In Apache Flink I have a stream of tuples. Let's assume a really simple Tuple1<String>
. The tuple can have an arbitrary value in it's value field (e.g. 'P1', 'P2', etc.). The set of possible values is finite but I don't know the full set beforehand (so there could be a 'P362'). I want to write that tuple to a certain output location depending on the value inside of the tuple. So e.g. I would like to have the following file structure:
-
/output/P1
-
/output/P2
/output/P1
/output/P2
在文档中,我只发现了可能写到我事先知道的位置(例如stream.writeCsv("/output/somewhere")
)的方法,但是没有办法让数据的内容决定数据的实际结束位置.
In the documentation I only found possibilities to write to locations that I know beforehand (e.g. stream.writeCsv("/output/somewhere")
), but no way of letting the contents of the data decide where the data is actually ending up.
我在文档中阅读了有关输出拆分的信息,但这似乎并没有提供一种将输出重定向到我想要的输出重定向到不同目标的方法(或者我只是不了解如何工作).
I read about output splitting in the documentation but this doesn't seem to provide a way to redirect the output to different destinations the way I would like to have it (or I just don't understand how this would work).
可以使用Flink API完成此操作吗,如果可以,如何执行?如果没有,也许有第三方图书馆可以做到这一点,还是我必须自己构建这样的东西?
Can this be done with the Flink API, if so, how? If not, is there maybe a third party library that can do it or would I have to build such a thing on my own?
推荐答案
您可以实现自定义接收器.从以下两者之一继承:
You can implement a custom sink. Inherit from one of both:
-
org.apache.flink.streaming.api.functions.sink.SinkFunction
-
org.apache.flink.streaming.api.functions.sink.RichSinkFunction
org.apache.flink.streaming.api.functions.sink.SinkFunction
org.apache.flink.streaming.api.functions.sink.RichSinkFunction
在程序中使用:
stream.addSink(SinkFunction<T> sinkFunction);
而不是stream.writeCsv("/output/somewhere")
.
这篇关于如何根据数据将一个数据流输出到不同的输出?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!