使用bq CLI在bigquery中列出数据集中的所有表，并将它们存储到Google云存储中 [英] List all the tables in a dataset in bigquery using bq CLI and store them to google cloud storage

问题描述

我在数据集中大约有108张桌子.我正在尝试使用以下bash脚本提取所有这些表:

I have around 108 tables in a dataset. I am trying to extract all those tables using the following bash script:

# get list of tables
tables=$(bq ls "$project:$dataset" | awk '{print $1}' | tail +3)

# extract into storage
for table in $tables
do
    bq extract --destination_format "NEWLINE_DELIMITED_JSON" --compression "GZIP" "$project:$dataset.$table" "gs://$bucket/$dataset/$table.json.gz" 
done

但是bq ls一次只能显示大约50张表，因此我无法将它们提取到云存储中.

But it seems that bq ls only show around 50 tables at once and as a result I can not extract them to cloud storage.

反正我可以使用bq ls命令访问所有108个表吗?

Is there anyway I can access all of the 108 tables using the bq ls command?

推荐答案

列出bq ls将显示的表时的默认行数是100.您可以使用命令行选项--max_results或-n进行更改.

The default number of rows when listing tables that bq ls will display is 100. You can change this with the command line option --max_results or -n.

您还可以在$HOME/.bigqueryrc中设置bq的默认值.

You can also set the default values for bq in $HOME/.bigqueryrc.

在.bigqueryrc中添加标志

这篇关于使用bq CLI在bigquery中列出数据集中的所有表，并将它们存储到Google云存储中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！