我们可以像在Hive中一样在Big Query中创建动态分区吗？ [英] Can we create dynamic partition in Big Query as in Hive?

问题描述

我希望存储在桶中的所有数据（例如：/bucket/project/odate/odate_cust.txt）能够在Big Query中的表中加载，该Big Query是DAY分区的？我是否需要逐一导入文件并加载，或者我可以直接加载到多个分区中。

  ** bq mk --time_partitioning_type = DAY market.cust custid：string，grp：integer，odate：string ** 
  

解决方案

目前，您需要指定分区（使用$ decorator语法）将每个负载放入相应的分区。否则，BigQuery将使用加载作业的UTC时间来选择分区。有一个即将推出的功能可以让你自己的字段进行分区（我假设你的文件中有一个TIMESTAMP或DATE字段可以分区）。不过，他们还没有推出（即将推出阿尔法）。您可以在此处追踪其进度。

<如果你很匆忙，那么有一些解决方法（例如，将其全部加载到非分区表中，然后使用SQL或云数据流对其进行分区）。看看这里。

最后，如果您的文件名包含该分区的日期/日期，那么您可以很容易地自行编写查看文件名并运行多个加载作业并将数据填充到表中的相应分区。

I wanted all my data stored in Buckets (ex: /bucket/project/odate/odate_cust.txt) to be loaded in the table in Big Query which is DAY partitioned? Do I need to import one by one file and load or I can load directly into multiple partition.

**bq mk --time_partitioning_type=DAY market.cust custid:string,grp:integer,odate:string**

解决方案

Currently, you would need to specify the partition (using the $ decorator syntax) for each load to put it in the corresponding partition. Otherwise, BigQuery will use the UTC time of the load job to select the partition. There's an upcoming feature that will allow partitioning by your own field (I assume you have either a TIMESTAMP or DATE field in your files that you can partition by). However, they have not rolled it out yet (it's going alpha soon). You can track its progress here.

If you are in a hurry, then there's a few workarounds (e.g. loading it all into a non-partitioned table, and then using SQL or Cloud Dataflow to partition it afterwards). Have a look here.

Finally, if your file names contain a date/day for the partition, then it would be easy enough to script something yourself that looks at the name of the file and runs multiple load jobs and stuffs the data into the corresponding partition in the table.

这篇关于我们可以像在Hive中一样在Big Query中创建动态分区吗？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！