使用Spark Big Query Connector的自定义SQL [英] Custom SQL using Spark Big Query Connector

问题描述

我有一些自定义sql可以从BigQuery读取数据.我该如何执行呢?我尝试使用选项作为查询，但无法正常工作.它会忽略查询选项并读取整个表.

I have some custom sql to read the data from BigQuery. How can I execute that? I tried using option as query but it is not working. It is ignoring the query option and reading the full table.

 Dataset<Row> testDS = session.read().format("bigquery")
                    //.option("table", <TABLE>)
                    .option("query",<QUERY>)
                    .option("project", <PROJECT_ID>)
                    .option("parentProject", <PROJECT_ID>)
                    .load();

推荐答案

这是因为 query 选项在连接器中不可用.参见 https://github.com/GoogleCloudDataproc/spark-bigquery-connector/README.md 获取完整的选项列表.

That's because the query option is not available in the connector. See https://github.com/GoogleCloudDataproc/spark-bigquery-connector/README.md for a full list of options.

您有几种选择:

• 使用您的自定义SQL创建视图，然后从视图中读取
• 使用查询结果创建一个临时表，读取这些结果，然后删除该表.

这篇关于使用Spark Big Query Connector的自定义SQL的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！