使用Google Cloud Dataflow如何在GCE Compute实例上以适当的凭据运行? [英] Using Google Cloud Dataflow how do I run with proper credentials on a GCE Compute instance?

问题描述

我是Google Cloud Dataflow的新手，这可能从下面的问题中显而易见.

I'm new to Google Cloud Dataflow, as is probably obvious from my questions below.

我已经编写了一个数据流应用程序，并且可以使用我的个人凭据在本地和GCE实例上运行它，而不会出现问题.但是，我似乎无法破解使用计算引擎实例的服务凭据或我使用API​​&所创建的服务凭据来使其运行的正确步骤.控制台的AUTH部分.我总是在运行时收到401未经授权的错误.

I've got a dataflow application written and can get it to run without issue using my personal credentials both locally and on a GCE instance. However, I can't seem to crack the proper steps to get it to run using the compute engine instance's service credentials or service credentials I've created using the API & AUTH section of the console. I always get a 401 not authorized error when I run.

这就是我尝试过的...

Here's what I've tried...

1)创建的虚拟机授予对存储，数据存储，sql和计算引擎的访问权限.我的理解是，这应该创建了一个特定于CI的服务帐户，该帐户是服务器的默认凭据.在此实例上运行应用程序时，应使用与使用用户身份验证相同的方式来使用它们.我在这里得到401.我的问题是...在哪里可以看到该服务帐户，该帐户原本应该创建?还是我只是依靠它存在于某处?

1) Created virtual machine granting access rights to storage, datastore, sql and compute engine. My understanding is that this supposedly created a CI specific services account that is the server's default credentials. These should be used the same way a user's authentication is used when an app is run on this instance. Here's where I get a 401. My question is... Where can I see this service account that was supposedly created? Or do I just rely that it exists somewhere?

2)在API&中创建服务凭证开发者控制台的Auth部分.然后使用cloud auth activate-service-account并通过将命令指向我下载的凭据json文件来激活该帐户.当您使用gcloud auth登录时，有点像OAUTH往返.在这里，我还得到了401.

2) Created service credentials in API & Auth portion of developer's console. Then used cloud auth activate-service-account and activated that account by pointing the command at the credentials json file I downloaded. Kind of like the OAUTH round trip when you use gcloud auth login. Here I also get the 401.

3)最后一件事是使用步骤2中的服务凭据(与GCE实例分开)，然后创建一个实现CredentialFactory接口的对象，并将其传递给PipelineOptions.但是，当应用运行时，该应用现在崩溃，并显示一条错误消息，指出它正在寻找fromCredentialFactory接口中没有的fromOptions方法.选项的配置方式，凭据工厂的外观以及由此产生的堆栈跟踪.

3) This last thing was using the service credentials from step 2 separate from the GCE instance and then create an object that implements the CredentialFactory interface and pass it off to the PipelineOptions. However, when it runs the app crashes now with an error saying that it is looking for a method, fromOptions, that isn't in the CredentialFactory interface. How the options were configured, what the credentials factory looked like and the stack trace from this follows.

如果我可以使用以上三种方法中的任何一种，我将很乐意使用上述三种方法中的任何一种来利用服务凭据.您可以提供关于我做错了什么，我遗漏的步骤以及其他未开发选项的任何见解.该文档有些脱节.如果有明确的分步指南，则指向该链接就足够了.到目前为止，我自己发现的东西几乎没有帮助.

I would be happy to utilize any of the above 3 methods to make use of service credentials, if I could get any of them to work. Any insight you can provide on what I'm doing wrong, steps I'm leaving out, other unexplored options would be greatly appreciated. The documentation is a little dis-jointed. If there is a clear step by step guide a link to that would be sufficient. What I've found so far on my own has been of little assistance.

如果我可以提供任何其他信息，请告诉我.

If I can provide any additional information please let me know.

有些代码可能会有所帮助，并且在使用凭据工厂运行代码时会得到堆栈跟踪.

Here's some code that may be helpful and the stack trace I get when the code runs using the credential factory.

选项设置代码如下:

GcrDataflowPipelineOptions options = PipelineOptionsFactory.fromArgs(args)
        .withValidation()
        .as(GcrDataflowPipelineOptions.class);
options.setKind("Counties");
options.setCredentialFactoryClass(GoogleCredentialProvider.class);

GoogleCredentialProvider.java

GoogleCredentialProvider.java

请注意，我在创建服务帐户(重命名)过程中下载的json文件是从我的应用类路径中作为资源加载的内容.

Notice the json file I downloaded as part of creating the services account (renamed) is what's loaded as a resource from my apps class path.

public class GoogleCredentialProvider implements CredentialFactory {

    @Override
    public Credential getCredential() throws IOException, GeneralSecurityException {
        final String env = System.getProperty("gcr_dataflow_env", "local");
        Properties props = new Properties();
        ClassLoader loader = this.getClass().getClassLoader();
        props.load(loader.getResourceAsStream(env + "-gcr-dataflow.properties"));
        final String credFileName = props.getProperty("gcloud.dataflow.service.account.file");
        InputStream credStream = loader.getResourceAsStream(credFileName);
        GoogleCredential credential = GoogleCredential.fromStream(credStream);
        return credential;
    }

}

Stacktrace:

Stacktrace:

java.lang.RuntimeException: java.lang.RuntimeException: Unable to find factory method com.scotcro.gcr.dataflow.components.pipelines.GoogleCredentialProvider#fromOptions
    at com.google.cloud.dataflow.sdk.runners.dataflow.BasicSerializableSourceFormat.evaluateReadHelper(BasicSerializableSourceFormat.java:268)
    at com.google.cloud.dataflow.sdk.io.Read$Bound$1.evaluate(Read.java:123)
    at com.google.cloud.dataflow.sdk.io.Read$Bound$1.evaluate(Read.java:120)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.visitTransform(DirectPipelineRunner.java:684)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:200)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:196)
    at com.google.cloud.dataflow.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:99)
    at com.google.cloud.dataflow.sdk.Pipeline.traverseTopologically(Pipeline.java:208)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.run(DirectPipelineRunner.java:640)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:354)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:76)
    at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:149)
    at com.scotcro.gcr.dataflow.app.GcrDataflowApp.run(GcrDataflowApp.java:65)
    at com.scotcro.gcr.dataflow.app.GcrDataflowApp.main(GcrDataflowApp.java:49)
Caused by: java.lang.RuntimeException: Unable to find factory method com.scotcro.gcr.dataflow.components.pipelines.GoogleCredentialProvider#fromOptions
    at com.google.cloud.dataflow.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
    at com.google.cloud.dataflow.sdk.util.InstanceBuilder.build(InstanceBuilder.java:161)
    at com.google.cloud.dataflow.sdk.options.GcpOptions$GcpUserCredentialsFactory.create(GcpOptions.java:180)
    at com.google.cloud.dataflow.sdk.options.GcpOptions$GcpUserCredentialsFactory.create(GcpOptions.java:175)
    at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:288)
    at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:127)
    at com.sun.proxy.$Proxy42.getGcpCredential(Unknown Source)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.getDatastore(DatastoreIO.java:335)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.createReader(DatastoreIO.java:320)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.createReader(DatastoreIO.java:186)
    at com.google.cloud.dataflow.sdk.runners.dataflow.BasicSerializableSourceFormat.evaluateReadHelper(BasicSerializableSourceFormat.java:259)
    ... 13 more
java.lang.RuntimeException: java.lang.RuntimeException: Unable to find factory method com.scotcro.gcr.dataflow.components.pipelines.GoogleCredentialProvider#fromOptions
2015-07-03 09:55:42,519 | main | DEBUG | co.sc.gc.da.ap.GcrDataflowApp | destroying
    at com.google.cloud.dataflow.sdk.runners.dataflow.BasicSerializableSourceFormat.evaluateReadHelper(BasicSerializableSourceFormat.java:268)
    at com.google.cloud.dataflow.sdk.io.Read$Bound$1.evaluate(Read.java:123)
    at com.google.cloud.dataflow.sdk.io.Read$Bound$1.evaluate(Read.java:120)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.visitTransform(DirectPipelineRunner.java:684)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:200)
    at com.google.cloud.dataflow.sdk.runners.TransformTreeNode.visit(TransformTreeNode.java:196)
    at com.google.cloud.dataflow.sdk.runners.TransformHierarchy.visit(TransformHierarchy.java:99)
    at com.google.cloud.dataflow.sdk.Pipeline.traverseTopologically(Pipeline.java:208)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner$Evaluator.run(DirectPipelineRunner.java:640)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:354)
    at com.google.cloud.dataflow.sdk.runners.DirectPipelineRunner.run(DirectPipelineRunner.java:76)
    at com.google.cloud.dataflow.sdk.Pipeline.run(Pipeline.java:149)
    at com.scotcro.gcr.dataflow.app.GcrDataflowApp.run(GcrDataflowApp.java:65)
    at com.scotcro.gcr.dataflow.app.GcrDataflowApp.main(GcrDataflowApp.java:49)
Caused by: java.lang.RuntimeException: Unable to find factory method com.scotcro.gcr.dataflow.components.pipelines.GoogleCredentialProvider#fromOptions
    at com.google.cloud.dataflow.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
    at com.google.cloud.dataflow.sdk.util.InstanceBuilder.build(InstanceBuilder.java:161)
    at com.google.cloud.dataflow.sdk.options.GcpOptions$GcpUserCredentialsFactory.create(GcpOptions.java:180)
    at com.google.cloud.dataflow.sdk.options.GcpOptions$GcpUserCredentialsFactory.create(GcpOptions.java:175)
    at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:288)
    at com.google.cloud.dataflow.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:127)
    at com.sun.proxy.$Proxy42.getGcpCredential(Unknown Source)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.getDatastore(DatastoreIO.java:335)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.createReader(DatastoreIO.java:320)
    at com.google.cloud.dataflow.sdk.io.DatastoreIO$Source.createReader(DatastoreIO.java:186)
    at com.google.cloud.dataflow.sdk.runners.dataflow.BasicSerializableSourceFormat.evaluateReadHelper(BasicSerializableSourceFormat.java:259)
    ... 13 more

推荐答案

您可能没有适当的凭据.当您从GCE执行Dataflow作业时，实例所附加的服务帐户将用于DataFlow的验证.

You likely do not have the proper credentials. When you execute a Dataflow job from GCE, The service account attached to the instance will be used for validation by DataFlow.

您在创建计算机时是否这样做?

Did you do this when creating your machines?

• 在GCE上为实例创建服务帐户? https://cloud.google.com/compute/docs/authentication#using

• create a service account for the instance on GCE? https://cloud.google.com/compute/docs/authentication#using

设置使用数据流所需的范围，例如存储，计算， 和bigquery? https://www.googleapis.com/auth/cloud-platform

Set the required scopes for using Dataflow such as storage, compute, and bigquery? https://www.googleapis.com/auth/cloud-platform

这篇关于使用Google Cloud Dataflow如何在GCE Compute实例上以适当的凭据运行?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！