谷歌云数据流使用的凭据 [英] credentials used by google cloud dataflow

问题描述

当前对数据流使用的凭据/配置有些困惑...

Currently have some confusion on the credentials/configuration used by dataflow...

根据我的实验，似乎数据流始终使用默认配置而不是活动配置.那是对的吗? (例如，在我的gcloud config中，如果我的活动配置在项目B上时我具有项目A的默认配置，则看来我的数据流作业将始终提交到项目A.同样，这种方式似乎是数据流作业是忽略options.setProject()中设置的内容，所以想知道何时再次使用options.getProject()进行数据流...?)

From my experimentation, it seems that dataflow is always using the default configuration instead of the active configuration. Is that correct? (For example in my gcloud config if I have a default configuration with project A while my active configuration is on project B, it seems that my dataflow job will submit to project A always. Also in this way it seems that the dataflow job is ignoring what is set in options.setProject(), so sort of wondering when is dataflow using options.getProject() again...?)

又想知道我是否可以通过自定义配置提交数据流作业，比如说我想在同一运行中以不同的凭据将多个作业提交到不同的项目(无需手动更改我的gcloud config)?

And also wondering is there any way that I submit dataflow job with customized configuration, say I want to submit multiple jobs to different projects with different credentials on the same run(without manually changing my gcloud config)?

btwbtw我正在数据流服务云平台上运行数据流作业，但如果有帮助，请从非gce Cloudservices帐户提交该作业.

btw I am running the dataflow job on dataflow services cloud platform but submit the job from non-gce Cloudservices Account if it will make a difference.

推荐答案

Google Cloud Dataflow默认使用应用程序默认凭据库，以获取未指定的凭据.该库当前仅支持使用gcloud默认配置获取凭据.同样，对于该项目，Google Cloud Dataflow使用gcloud默认配置.

Google Cloud Dataflow by default uses the application default credentials library to get the credentials if they are not specified. The library currently only supports getting the credentials using the gcloud default configuration. Similarly, for the project, Google Cloud Dataflow uses the gcloud default configuration.

要能够在其他项目中运行作业，可以在命令行上手动指定(例如，使用

To be able to run jobs with a different project, one can manually specify on the command-line (for example --project=myProject, if using PipelineOptionsFactory.fromArgs) or set the option explicitly utilizing GcpOptions.setProject.

为了能够使用不同的凭据运行作业，可以构建应用程序默认凭据库支持的方式自动生成与Google Cloud Dataflow绑定在一起的凭据对象.一个示例是按照此处.

To be able to run jobs with different credentials, one can construct a credentials object and can explicitly set it utilizing GcpOptions.setGcpCredential or one can rely on using the ways that the application default credentials library supports generating the credentials object automatically which Google Cloud Dataflow is tied into. One example would be to use the environment variable GOOGLE_APPLICATION_CREDENTIALS as explained here.

这篇关于谷歌云数据流使用的凭据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！