如何在谷歌云数据流中运行动态第二查询? [英] How to run dynamic second query in google cloud dataflow?
本文介绍了如何在谷歌云数据流中运行动态第二查询?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在尝试执行一项操作,其中我通过查询获取 Id 列表,将它们转换为以逗号分隔的字符串(即1,2,3"),然后在辅助查询中使用它.尝试运行第二个查询时,出现语法错误:
I'm attempting to do an operation wherein I get a list of Ids via a query, transform them into a string separated by commas (i.e. "1,2,3") and then use it in a secondary query. When attempting to run the second query, I'm given a syntax error:
lambda 转换的目标类型必须是接口"
"Target type of a lambda conversion must be an interface"
String query = "SELECT DISTINCT campaignId FROM `" + options.getEligibilityInputTable() + "` ";
Pipeline p = Pipeline.create(options);
p.apply("GetCampaignIds", BigQueryIO.readTableRows().withTemplateCompatibility().fromQuery(query).usingStandardSql())
.apply("TransformCampaignIds",
MapElements.into(TypeDescriptors.strings())
.via((TableRow row) -> (String)row.get("campaignId")))
.apply(Combine.globally(new StringToCsvCombineFn()))
.apply("GetAllCampaigns", campaignIds -> BigQueryIO.readTableRows().withTemplateCompatibility().fromQuery("SELECT id AS campaignId, dataQuery FROM `{projectid}.mysql_standard.campaigns` WHERE campaignId IN (" + campaignIds + ")").usingStandardSql())
....
如何将查询链接在一起?
How can I chain queries together?
推荐答案
很遗憾,您无法使用现有的资源来做到这一点.您有两种选择:
You cannot do this with the existing sources, unfortunately. Your options here are two:
- 您从 ParDo 手动调用 BQ API.
- 您编写了一个复杂的 SQL 查询来为您执行此操作.
第二个选项看起来像这样:
The second option looks something like so:
String query = "SELECT id AS campaignId, dataQuery \
FROM `{projectid}.mysql_standard.campaigns` \
WHERE campaignId IN ( \
SELECT DISTINCT campaignId \
FROM `" + options.getEligibilityInputTable()
+ "`)";
Pipeline p = Pipeline.create(options);
p.apply("GetAllCampaigns", BigQueryIO.readTableRows()
.withTemplateCompatibility()
.fromQuery(query)
.usingStandardSql());
这篇关于如何在谷歌云数据流中运行动态第二查询?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文