从Google BigQuery导出到CloudSQL？ [英] Export from Google BigQuery into CloudSQL?

问题描述

我希望将BigQuery中的数据（即报告）转储到CloudSQL数据库中，以编程方式实现此目的的最佳方法是什么？ 我知道我可以做到这一点通过运行BigQuery查询，手动 ，将其下载为CSV格式，然后通过云控制台上传它，但我想以编程方式执行此操作，最好是在Python / SQL中。 解决方案

就像转储整个表一样，您可以使用BigQuery和Cloud SQL API的组合来实现此目的。

BigQuery文档在python中有一个API示例，用于将BigQuery表提取到云存储。

将数据存入云端存储后，您可以使用 Cloud SQL Admin API 将数据导入MySQL表。

如果您需要更精细的控制，可以使用BigQuery API执行查询，获取结果，连接到Cloud SQL实例并插入数据。如果数据量很大，这将不会很好。

更复杂的方法是使用Dataflow将您感兴趣的数据写入云存储，然后使用Cloud SQL API导入它。

（为了我自己的好奇心，你能描述一下在Cloud SQL而不是BigQuery中使用数据的用例吗？我/我们了解我们的客户如何使用我们的产品以及我们可以改进的地方。）

I want to dump data from BigQuery (i.e. reports) into a CloudSQL database, what is the best way to achieve this programatically?

I realise I could do this manually by running a BigQuery query, downloading it as a CSV, then uploading it through the Cloud console, but I want to do this programatically, preferably in Python/SQL.

解决方案

If you would like to dump entire tables, you can use a combination of the BigQuery and Cloud SQL APIs to achieve this.

The BigQuery documentation has an API example in python for extracting a BigQuery table to Cloud Storage.

Once the data is in Cloud Storage, you can use the Cloud SQL Admin API to import the data into a MySQL table.

If you need more granular control, you can use the BigQuery API to perform the query, fetch the results, connect to the Cloud SQL instance and insert the data. This won't perform as well if the amount of data is large.

A more complex approach is to use Dataflow to write the data you are interested in to Cloud Storage and use the Cloud SQL API to import it.

(For my own curiosity, can you describe the use case for wanting the data in Cloud SQL instead of BigQuery? It will help me/us understand how our customers are using our product and where we can improve.)

这篇关于从Google BigQuery导出到CloudSQL？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！