Google Dataflow作业和BigQuery在不同地区失败 [英] Google Dataflow job and BigQuery failing on different regions

问题描述

  BigQuery工作...完成了错误（s）： errorResult：
无法在不同位置读写：源：EU，目标：US，错误：无法在不同位置读写：源：EU，目的地：US 
  

我以
- zone = europe-west1-b

这是管道中唯一能处理BigQuery的部分：

  Pipeline p = Pipeline.create（options）; 
 p.apply（BigQueryIO.Read.fromQuery（query））; 
  

我正在阅读的BigQuery表具有以下细节： Data位置EU

当我在本地运行作业时，我得到：

  SEVERE：打开数据集的BigQuery表dataflow_temporary_table_339775时出错_dataflow_temporary_dataset_744662：404未找到
  

我不明白为什么它试图写入不同的位置，如果我只是读数据。即使它需要创建一个临时表，为什么它是在另一个区域创建的？

任何想法？

位于同一区域。

如果Dataflow中使用的Google云端存储位置也位于同一个区域中。

I have a Google Dataflow job that is failing on:

BigQuery job ... finished with error(s): errorResult: 
Cannot read and write in different locations: source: EU, destination: US, error: Cannot read and write in different locations: source: EU, destination: US

I'm starting the job with --zone=europe-west1-b

And this is the only part of the pipeline that does anything with BigQuery:

Pipeline p = Pipeline.create(options);
p.apply(BigQueryIO.Read.fromQuery(query));

The BigQuery table I'm reading from has this in the details: Data Location EU

When I run the job locally, I get:

SEVERE: Error opening BigQuery table  dataflow_temporary_table_339775 of dataset _dataflow_temporary_dataset_744662  : 404 Not Found

I don't understand why it is trying to write to a different location if I'm only reading data. And even if it needs to create a temporary table, why is it being created in a different region?

Any ideas?

解决方案

I would suggest to verify:

• If the staging location for the Google Dataflow is in the same zone.
• If Google Cloud Storage location used in Dataflow is also the in same zone.

这篇关于Google Dataflow作业和BigQuery在不同地区失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！