如何在运行时从flink中的datastream的map或flatMap执行sql查询 [英] how to execute sql queries at run time from datastream's map or flatMap in flink

问题描述

我是flink的新手，并且想了解如何使用FLINK运行我的用例: 应用程序具有三个输入数据源 a)历史数据 b)从kafka获取所有直播活动 c)获取具有触发条件的控制事件

I am new to flink and want to understand how to run my use case with FLINK: Application has three input data source a) historical data b) get all the live events from kafka c) get the control event that will have a trigger condition

由于该应用程序正在处理历史数据，所以我认为我将合并历史数据和实时数据，并在该流上创建一个表.

since the application is dealing with historical data so I thought that I will merge historical data and live data and will create a table on that stream.

要触发该事件，我们必须在作为输入源并保存where子句的控制事件的帮助下编写SQL查询.

To trigger the event we have to write the SQL query with help of control event that is the input source and that holds the where clause.

我的问题是，当数据在Stream中以及执行类似操作时，构建SQL查询

My problem is to build the SQL query as data is in Stream and when I do something like

DataStream<ControlEvent> controlEvent
controlEvent.map(new FlatMapFunction(String, String)
{
   @override
   public String flatMap(String s, Collector<String> coll)
   {
     tableEnv.execute("select * from tableName");   /// throw serialization exception
   }
});

它不会引发序列化异常Localexecutionenvironment

it throws not serialization exception Localexecutionenvironment

推荐答案

Flink SQL尚不支持这种动态查询注入.

That sort of dynamic query injection is not (yet) supported by Flink SQL.

更新:

考虑到您所说的需求(即查询中的变化将受到限制)，您可能要做的是使用DataStream API而不是SQL来实现.这可能是KeyedBroadcastProcessFunction，它将保留某些键控状态，并且您可以在查询/查询的更新中广播.

Given what you've said about your requirements -- that the variations in the queries will be limited -- what you might do instead is to implement this using the DataStream API, rather than SQL. This would probably be a KeyedBroadcastProcessFunction that would hold some keyed state and you could broadcast in the updates to the query/queries.

看看欺诈检测演示作为示例，说明如何使用Flink构建此类事物.

Take a look at the Fraud Detection Demo as an example of how to build this sort of thing with Flink.

这篇关于如何在运行时从flink中的datastream的map或flatMap执行sql查询的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！