将 10 个数据集(每个数据集有 80 个表)从 bigquery 导出到 google 存储的有效方法? [英] An efficient way of exporting 10 datasets (each having 80 tables) from bigquery to google storage?

问题描述

I have 10 datasets in BigQuery and each dataset has 80 tables. I know I can export each table in each dataset one-by-one to google storage using console or Web UI. This is for back-up purposes. However, this would take a while.

I wonder if there is a more handy way to handle this?

解决方案

You can do it using bq command line tools (https://cloud.google.com/bigquery/bq-command-line-tool)

Following code will help you to achieve the same:

project="bigquery-project"
dataset="your-dataset"
bucket="your-bucket"

# get list of tables
tables=$(bq ls "$project:$dataset" | awk '{print $1}' | tail +3)

for table in $tables
do
    bq extract --destination_format "CSV" --compression "GZIP" "$project:$dataset.$table" "gs://$bucket/$table/$table.csv.gz" 
done

Additionally, if you want to run command in async mode replace extract command with following

    bq --nosync extract --destination_format "CSV" --compression "GZIP" "$project:$dataset.$table" "gs://$bucket/$table/$table.csv.gz" 

PS:

• Make sure that bq command line tools is installed and it is in your environment. It comes with google cloud SDK. To verify you can run bq help

这篇关于将 10 个数据集(每个数据集有 80 个表)从 bigquery 导出到 google 存储的有效方法?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！