无法写入bigquery-权限被拒绝:Apache Beam Python-Google Dataflow [英] Unable to Write to bigquery - Permission denied: Apache Beam Python - Google Dataflow

问题描述

我已经使用谷歌云数据流服务使用apache beam python sdk了一段时间了.

I have been using apache beam python sdk using google cloud dataflow service for quite some time now.

我正在为一个新项目设置数据流.

I was setting dataflow up for a new project.

数据流管道

1. 从Google数据存储读取数据
2. 处理
3. 写给Google Big-Query.

我在其他运行良好的项目上也有类似的管道.

I have similar pipelines running on other projects which are running perfectly fine.

今天，当我开始执行数据流作业时，管道启动了，从数据存储中读取数据，进行了处理，并在将其写入bigquery时导致了

Today, When I started a dataflow job, the pipeline started, read data from datastore, processed it and when it was about to write it to bigquery, It resulted in

apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: 
Dataflow pipeline failed. State: FAILED, Error:
Workflow failed. Causes: S04:read from datastore/GroupByKey/Read+read 
from datastore/GroupByKey/GroupByWindow+read from datastore/Values+read 
from datastore/Flatten+read from datastore/Read+convert to table 
rows+write to bq/NativeWrite failed., BigQuery import job 
"dataflow_job_8287310405217525944" failed., BigQuery creation of import 
job for table "TableABC" in dataset "DatasetABC" in project "devel- 
project-abc" failed., BigQuery execution failed., Error:
Message: Access Denied: Dataset devel-project-abc:DatasetABC: The user 
[e-mail] does not 
have bigquery.tables.create permission for dataset devel-project- 
abc:DatasetABC: HTTP Code: 403

我确保所有必需的API均已启用.据我说，该服务帐户具有必要的许可.

I made sure all the required API are enabled. According to me the service account has the necessary permission.

我的问题是这可能是哪里出了错?

My question is Where this might be going wrong?

更新

根据我在以前的项目(准确地说是3个不同的项目)中所记得的内容，我没有给数据流服务代理任何特定的许可. 计算引擎服务代理具有数据流管理员，编辑器，数据流查看器之类的权限. 因此，在继续授予服务代理与bigquery相关的权限之前，我想知道为什么环境的行为方式与以前的项目不同.

From what I remember on previous projects (3 different projects to be precise) I didn't give the dataflow service agent any specific permission. The compute engine service agent had permissions like dataflow admin, editor, dataflow viewer. Hence before proceeding with giving the service agent permissions related to bigquery, i would like to know why the environment is behaving differently than the previous projects.

最近几个月上线的许可/政策变更/更新是否存在，导致需要bigquery writer许可?

Is there any permission/policy changes/updates that went live in last few months resulting in requirement of bigquery writer permission?

推荐答案

请确保您的服务帐户('[e-mail]')在其中具有'roles/bigquery.dataEditor'角色'devel-project-abc:DatasetABC'. 另外，请确保为您的项目启用了"BigQuery Data Editor"角色.

Please make sure your service account ('[e-mail]') has 'roles/bigquery.dataEditor' role in 'devel-project-abc:DatasetABC'. Also make sure 'BigQuery Data Editor' role is enabled for your project.

GCP IAM 中进行检查.

这篇关于无法写入bigquery-权限被拒绝:Apache Beam Python-Google Dataflow的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！