如何在运行时从数据流的地图或 flink 中的 flatMap 执行 sql 查询 [英] how to execute sql queries at run time from datastream's map or flatMap in flink

问题描述

我是 flink 的新手，想了解如何使用 FLINK 运行我的用例:应用程序有三个输入数据源a) 历史数据b) 从 kafka 获取所有实时事件c) 获取将有触发条件的控制事件

I am new to flink and want to understand how to run my use case with FLINK: Application has three input data source a) historical data b) get all the live events from kafka c) get the control event that will have a trigger condition

由于应用程序正在处理历史数据，所以我认为我将合并历史数据和实时数据，并在该流上创建一个表.

since the application is dealing with historical data so I thought that I will merge historical data and live data and will create a table on that stream.

要触发事件，我们必须借助作为输入源并包含 where 子句的控制事件编写 SQL 查询.

To trigger the event we have to write the SQL query with help of control event that is the input source and that holds the where clause.

我的问题是构建 SQL 查询，因为数据在 Stream 中，当我执行类似操作时

My problem is to build the SQL query as data is in Stream and when I do something like

DataStream<ControlEvent> controlEvent
controlEvent.map(new FlatMapFunction(String, String)
{
   @override
   public String flatMap(String s, Collector<String> coll)
   {
     tableEnv.execute("select * from tableName");   /// throw serialization exception
   }
});

它抛出非序列化异常 Localexecutionenvironment

it throws not serialization exception Localexecutionenvironment

推荐答案

Flink SQL(尚)不支持这种动态查询注入.

That sort of dynamic query injection is not (yet) supported by Flink SQL.

更新:

鉴于您所说的关于您的要求——查询中的变化将受到限制——您可能会做的是使用 DataStream API 而不是 SQL 来实现这一点.这可能是一个 KeyedBroadcastProcessFunction，它将保存一些键控状态，您可以在查询/查询的更新中广播.

Given what you've said about your requirements -- that the variations in the queries will be limited -- what you might do instead is to implement this using the DataStream API, rather than SQL. This would probably be a KeyedBroadcastProcessFunction that would hold some keyed state and you could broadcast in the updates to the query/queries.

查看欺诈检测演示 作为如何使用 Flink 构建此类东西的示例.

Take a look at the Fraud Detection Demo as an example of how to build this sort of thing with Flink.

这篇关于如何在运行时从数据流的地图或 flink 中的 flatMap 执行 sql 查询的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！