Google Dataflow 作业和 BigQuery 在不同区域失败 [英] Google Dataflow job and BigQuery failing on different regions

问题描述

我有一个 Google Dataflow 作业失败:

I have a Google Dataflow job that is failing on:

BigQuery job ... finished with error(s): errorResult: 
Cannot read and write in different locations: source: EU, destination: US, error: Cannot read and write in different locations: source: EU, destination: US

我开始工作--zone=europe-west1-b

这是管道中唯一可以使用 BigQuery 执行任何操作的部分:

And this is the only part of the pipeline that does anything with BigQuery:

Pipeline p = Pipeline.create(options);
p.apply(BigQueryIO.Read.fromQuery(query));

我正在阅读的 BigQuery 表的详细信息如下:Data Location EU

The BigQuery table I'm reading from has this in the details: Data Location EU

当我在本地运行作业时，我得到:

When I run the job locally, I get:

SEVERE: Error opening BigQuery table  dataflow_temporary_table_339775 of dataset _dataflow_temporary_dataset_744662  : 404 Not Found

如果我只是在读取数据，我不明白为什么它会尝试写入不同的位置.而且即使需要创建临时表，为什么要创建在不同的区域?

I don't understand why it is trying to write to a different location if I'm only reading data. And even if it needs to create a temporary table, why is it being created in a different region?

有什么想法吗?

推荐答案

我建议验证:

• 如果 Google 数据流的暂存位置在同一区域中.
• 如果 Dataflow 中使用的 Google Cloud Storage 位置也在同一地区.

这篇关于Google Dataflow 作业和 BigQuery 在不同区域失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！