无法写入 bigquery - 权限被拒绝:Apache Beam Python - Google Dataflow [英] Unable to Write to bigquery - Permission denied: Apache Beam Python - Google Dataflow

问题描述

我已经使用谷歌云数据流服务使用 apache beam python sdk 已经有一段时间了.

I have been using apache beam python sdk using google cloud dataflow service for quite some time now.

我正在为一个新项目设置数据流.

I was setting dataflow up for a new project.

数据流管道

1. 从谷歌数据存储区读取数据
2. 处理它
3. 写入 Google Big-Query.

我在其他运行良好的项目上运行了类似的管道.

I have similar pipelines running on other projects which are running perfectly fine.

今天，当我开始一个数据流作业时，管道启动，从数据存储读取数据，处理它，当它准备将它写入 bigquery 时，结果是

Today, When I started a dataflow job, the pipeline started, read data from datastore, processed it and when it was about to write it to bigquery, It resulted in

apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: 
Dataflow pipeline failed. State: FAILED, Error:
Workflow failed. Causes: S04:read from datastore/GroupByKey/Read+read 
from datastore/GroupByKey/GroupByWindow+read from datastore/Values+read 
from datastore/Flatten+read from datastore/Read+convert to table 
rows+write to bq/NativeWrite failed., BigQuery import job 
"dataflow_job_8287310405217525944" failed., BigQuery creation of import 
job for table "TableABC" in dataset "DatasetABC" in project "devel- 
project-abc" failed., BigQuery execution failed., Error:
Message: Access Denied: Dataset devel-project-abc:DatasetABC: The user 
[e-mail] does not 
have bigquery.tables.create permission for dataset devel-project- 
abc:DatasetABC: HTTP Code: 403

我确保启用了所有必需的 API.根据我的说法，服务帐户具有必要的权限.

I made sure all the required API are enabled. According to me the service account has the necessary permission.

我的问题是这可能是哪里出了问题?

My question is Where this might be going wrong?

更新

根据我在以前的项目中的记忆(准确地说是 3 个不同的项目)，我没有给数据流服务代理任何特定的权限.计算引擎服务代理具有数据流管理员、编辑器、数据流查看器等权限.因此，在继续授予与 bigquery 相关的服务代理权限之前，我想知道为什么环境的行为与以前的项目不同.

From what I remember on previous projects (3 different projects to be precise) I didn't give the dataflow service agent any specific permission. The compute engine service agent had permissions like dataflow admin, editor, dataflow viewer. Hence before proceeding with giving the service agent permissions related to bigquery, i would like to know why the environment is behaving differently than the previous projects.

是否有任何权限/政策更改/更新在过去几个月生效，导致需要 Bigquery writer 权限?

Is there any permission/policy changes/updates that went live in last few months resulting in requirement of bigquery writer permission?

推荐答案

请确保您的服务帐户 ('[e-mail]') 具有 'roles/bigquery.dataEditor' 角色'devel-project-abc:DatasetABC'.还要确保为您的项目启用了BigQuery Data Editor"角色.

Please make sure your service account ('[e-mail]') has 'roles/bigquery.dataEditor' role in 'devel-project-abc:DatasetABC'. Also make sure 'BigQuery Data Editor' role is enabled for your project.

GCP IAM 是您可以检查的地方.

GCP IAM is where you can check those.

这篇关于无法写入 bigquery - 权限被拒绝:Apache Beam Python - Google Dataflow的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！