BigQuery:如何使用新的存储JSON文件自动重新加载表格? [英] BigQuery: How to autoreload table with new storage JSON files?

问题描述

我刚刚通过链接Google Cloud Storage中的可用JSON文件创建了一个BigQuery表.但是我看不到任何选项可以自动重载表格行，并在Google Cloud Storage文件夹或存储桶中添加新文件.

当前，我必须转到BigQuery控制台，然后删除&重新创建相同的表以加载新文件.但是该解决方案对我们来说是不可扩展的，因为我们在BigQuery API上运行了cron作业.如何在BigQuery中自动重新加载数据?

谢谢

解决方案

在Google Cloud Storage中的文件顶部定义外部表时-您可以使用通配符作为源位置，因此表将代表所有匹配的文件

然后，当您查询此类表时-您可以使用_file_name字段来告诉"给定行来自哪个文件

  SELECT_file_name AS文件，*来自`yourTable` 

这种方式-每当您在GCS中添加新文件时-都会自动"在表中获取它

I have just created one BigQuery table by linking available JSON files in Google Cloud Storage. But I do not see any option to auto-reload table rows with new files added in Google Cloud Storage folder or bucket.

Currently, I have to go to BigQuery console and then delete & recreate the same table to load new files. But this solution is not scalable for us because we run a cron job on BigQuery API. How to auto-reload data in BigQuery?

Thanks

解决方案

When you define External Table on top of Files in Google Cloud Storage - you can use wildcard for Source Location, so your table will represent all files that match

Then, when you query such table - you can use _file_name field which will "tell" you which file given row came from

SELECT  
  _file_name AS file,
  *
FROM `yourTable`   

This way - whenever you add new file in GCS - you will get it in table "automatically"

这篇关于BigQuery:如何使用新的存储JSON文件自动重新加载表格?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！