每秒超出tabledata.list字节的配额 [英] Exceeded quota for tabledata.list bytes per second

问题描述

我在BigQuery数据集中的表上触发了一组查询.

I am firing a set of queries on tables in BigQuery dataset.

共有3个select *查询，如下所示:

There are 3 select * queries as below:

"Select * from table1"    //1.3M records and 2.5GB data
"Select * from table2"      //0.3M records and 15 GB data
"Select * from table3"     //2M    records and 3GB data

我们正在使用spark连接器查询以上表格.但是，间歇性地我们看到一个错误:

We are querying the above tables using the spark connector. However intermittently we are seeing an error:

403禁止"
"domain":"usageLimits"，
message:" message:"超出了速率限制:您的项目:每个项目每秒的tabledata.list字节数超出配额.

403 Forbidden"
"domain" : "usageLimits",
message: " "message" : "Exceeded rate limits: Your project: exceeded quota for tabledata.list bytes per second per project.

这里的假设是tabledata列表调用失败，因为它每秒返回60 MB以上，这似乎是 https://cloud.google.com/bigquery/troubleshooting-errors

The assumption here is that the tabledata list call is failing because it is returning more than 60 MB per sec which seems to be the default quota as per https://cloud.google.com/bigquery/troubleshooting-errors

推荐答案

Tabledata.list并未真正针对Spark等高吞吐量用例进行优化.您可能想查看其他选项以读取BigQuery，特别是原生Spark连接器.

Tabledata.list isn't really optimized for high-throughput use cases such as Spark. You may want to check out other options for reading from BigQuery -- in particular, this use case is what the BigQuery Storage API is designed for, including a native Spark connector.

这篇关于每秒超出tabledata.list字节的配额的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！