在Google Cloud dataproc中使用非默认服务帐户 [英] Using non-default service account in Google Cloud dataproc

问题描述

我想创建一个在非默认服务帐户下运行的数据集群。计算实例的以下工作：

  gcloud compute instances create instance-1 --machine-typen1-standard-1 --zoneeurope-west1-b--scopes [e-mail]=https://www.googleapis.com/auth/cloud-platform
  

但创建dataproc实例时，同样的 - scopes 参数失败：

  gcloud数据集群集群创建--zoneeurope-west1-b--scopes [e-mail]=https ：//www.googleapis.com/auth/cloud-platformtestdataproc12345 
  

错误：（gcloud.dataproc.clusters.create）无效的服务帐户范围：'[e-mail]= https://www.googleapis.com/auth/cloud-platform '

是有可能在非默认服务帐户下运行dataproc？

不幸的是，目前没有办法使用正常的范围和元数据 - 介导的身份验证设置来指定自定义服务帐户。这是一个已知的功能请求，所以它应该可用于未来的Dataproc更新。

与此同时，即使您不能禁用在使用Dataproc时，使用默认GCE服务帐户的存储器读/写范围，您可以使用 IAM&管理员>服务帐户页面以获取服务帐户的JSON密钥文件，然后执行两项操作： 在集群创建时添加以下属性：

   - 属性core：fs.gs.auth.service.account.json。 keyfile = / etc / hadoop / conf / my-service-account.json 
  

使用将您的JSON密钥文件复制到您的节点的init动作;请注意，这仍然意味着您的JSON密钥文件必须可以作为阅读器访问GCE默认服务帐户，并且有权访问JSON密钥文件的GCS位置的任何人也可以代表该服务帐户进行操作，因此您仍然需要保持项目的安全。

 ＃！/ bin / bash 
＃将此处保存为gs ：//somepath/my-keyfile-setup.sh 
 
 gsutil cp gs：//path/to/your/json/file/in/gcs/my=service-account.json \ 
 /etc/hadoop/conf/my-service-account.json 
  

然后应用init动作：

  gcloud dataproc集群创建 - 初始化 - 动作gs：//somepath/my-keyfile-setup.sh。 .. 
  

I'd like to create a dataproc cluster that runs under a non-default service account. The following works for a compute instance:

gcloud compute instances create instance-1 --machine-type "n1-standard-1" --zone "europe-west1-b" --scopes [e-mail]="https://www.googleapis.com/auth/cloud-platform"

But the same --scopes argument fails when creating a dataproc instance:

gcloud dataproc clusters create --zone "europe-west1-b" --scopes [e-mail]="https://www.googleapis.com/auth/cloud-platform" testdataproc12345

ERROR: (gcloud.dataproc.clusters.create) Invalid service account scope: '[e-mail]=https://www.googleapis.com/auth/cloud-platform'

Is it possible to run dataproc under a non-default service account?

解决方案

Unfortunately, at the moment there's no way to specify your custom service accounts using the normal "scopes and metadata"-mediated auth setup. This is a known feature request, however, so it should become available in a future Dataproc update.

In the meantime, even though you can't disable the existence of the "storage read/write" scope with the default GCE service account when using Dataproc, you can make the Hadoop side use a particular service account via keyfiles by using the "Create Key" option under the IAM & Admin > Service accounts page to obtain a JSON keyfile for your service account, and then do two things:

1. Add the following property at cluster creation time:

   --properties core:fs.gs.auth.service.account.json.keyfile=/etc/hadoop/conf/my-service-account.json
   

2. Use an init action which copies your JSON keyfile to your nodes; note that this still means your JSON keyfile must be accessible to the GCE default service account as a reader, and anyone who has access to the GCS location of your JSON keyfile also has the ability to now act on behalf of that service account, so you still need to keep your project secure as necessary.

   #!/bin/bash
   # Save this somewhere as gs://somepath/my-keyfile-setup.sh
   
   gsutil cp gs://path/to/your/json/file/in/gcs/my=service-account.json \
       /etc/hadoop/conf/my-service-account.json
   

   And then apply that init action:

   gcloud dataproc clusters create --initialization-actions gs://somepath/my-keyfile-setup.sh ...
   

这篇关于在Google Cloud dataproc中使用非默认服务帐户的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！