将多个文件从Cloud Storage导入到BigQuery的命令 [英] Command to import multiple files from Cloud Storage into BigQuery

问题描述

我发现此命令列出了所有文件的路径:

I've figured that this command lists paths to all files:

gsutil ls "gs://bucket/foldername/*.csv"

此命令将文件导入BQ并自动检测模式:

This command imports a file to BQ and autodetects schema:

bq load --autodetect --source_format=CSV dataset.tableName gs://bucket/foldername/something.csv

现在，我需要使其协同工作以将所有文件导入BQ中的相应表.如果存在表，则将其替换.你能帮我一下吗?

Now I need to make it work together to import all files to respective tables in BQ. If table exists, then replace it. Could you give me a hand?

推荐答案

首先，创建一个包含所有列表的文件，其中包含要加载到BigQuery中的所有文件夹:

First, create a file with all the list with all the folders you want to load into BigQuery:

gsutil ls "gs://bucket/foldername/*.csv" > allmynicetables.txt

然后，创建一个简单的循环以对allmynicetables.txt上列出的每个csv文件重复加载操作:

Then, create a simple loop to repeat the load operation for every csv file listed on allmynicetables.txt:

while read p ; do bq load --autodetect --replace=true --source_format=CSV dataset.tableName $p ; done < allmynicetables.txt

只需澄清一下

-replace = true 可以覆盖现有表.

此外，我不确定为什么要放入 dataset.tableName ，是否始终将其复制到同一数据集?您可以从.csv源文件的名称中提取所需的数据集/表名称吗?您的问题对我来说还不清楚，请澄清.

Also, I am not sure why you put dataset.tableName, are you always copying to the same dataset? Can you extract the desired dataset/table name from the name of your .csv source file? This is not clear to me from your question, please clarify.

这篇关于将多个文件从Cloud Storage导入到BigQuery的命令的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！