Eclipse 上带有 Dataflow Runner 的 Apache Beam MinimalWordcount 示例 [英] Apache Beam MinimalWordcount example with Dataflow Runner on eclipse

问题描述

我正在尝试使用来自 Eclipse 的 DataFlowRunner 在 Windows 上运行 MinimalWordCount 示例，使用 MinimalWordCount -->Run As Java Application from with in eclipse ，它与使用我的 gcs 存储桶的示例中的股票代码相同，但是我始终得到以下异常，有人可以让我知道这里有什么问题吗?

1. 我已验证存储桶名称正确.
2. 我已经在我的 Windows 机器上运行了 gcloud init.

<块引用>

线程main"中的异常 java.lang.RuntimeException: Failed to从工厂方法构造实例DataflowRunner#fromOptions(接口org.apache.beam.sdk.options.PipelineOptions)在 org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233)在 org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)在 org.apache.beam.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:56)在 org.apache.beam.sdk.Pipeline.create(Pipeline.java:135)在 com.dynaobject.apachebeamexperiment.MinimalWordCount.main(MinimalWordCount.java:77)引起:java.lang.reflect.InvocationTargetException在 sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)在 sun.reflect.NativeMethodAccessorImpl.invoke(来源不明)在 sun.reflect.DelegatingMethodAccessorImpl.invoke(未知来源)在 java.lang.reflect.Method.invoke(Unknown Source)在 org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)... 4个引起:java.lang.IllegalArgumentException:DataflowRunner 需要 gcpTempLocation，但无法从中检索值管道选项在 org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:212)... 9 更多引起:java.lang.IllegalArgumentException:为 gcpTempLocation 构建默认值时出错:tempLocation 不是一个有效的 GCS 路径，gs://tempxyz.在 org.apache.beam.sdk.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:219)在 org.apache.beam.sdk.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:205)在 org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:575)在 org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:516)在 org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:155)在 com.sun.proxy.$Proxy15.getGcpTempLocation(来源不明)在 org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:210)... 9 更多由:java.lang.IllegalArgumentException:输出路径不存在或不可写:gs://tempxyz在 org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:191)在 org.apache.beam.sdk.util.GcsPathValidator.verifyPathIsAccessible(GcsPathValidator.java:78)在 org.apache.beam.sdk.util.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:62)在 org.apache.beam.sdk.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:217)……还有 15 个

解决方案

文档中的某处提到我需要运行以下内容.

<块引用>

gcloud auth 应用-默认登录

显然异常无助于解决这个问题，我希望它可以帮助其他人.

I am trying to run the MinimalWordCount example using the DataFlowRunner from Eclipse on windows using MinimalWordCount -->Run As Java Application from with in eclipse , its the same stock code from the example using my gcs bucket , however I consistently get the following exception , can some one let me know whats the issue here?

1. I have verified that the bucket name is correct.
2. I already ran the gcloud init on my Windows machine.

Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233) at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162) at org.apache.beam.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:56) at org.apache.beam.sdk.Pipeline.create(Pipeline.java:135) at com.dynaobject.apachebeamexperiment.MinimalWordCount.main(MinimalWordCount.java:77) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222) ... 4 more Caused by: java.lang.IllegalArgumentException: DataflowRunner requires gcpTempLocation, but failed to retrieve a value from PipelineOptions at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:212) ... 9 more Caused by: java.lang.IllegalArgumentException: Error constructing default value for gcpTempLocation: tempLocation is not a valid GCS path, gs://tempxyz. at org.apache.beam.sdk.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:219) at org.apache.beam.sdk.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:205) at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:575) at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:516) at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:155) at com.sun.proxy.$Proxy15.getGcpTempLocation(Unknown Source) at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:210) ... 9 more Caused by: java.lang.IllegalArgumentException: Output path does not exist or is not writeable: gs://tempxyz at org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:191) at org.apache.beam.sdk.util.GcsPathValidator.verifyPathIsAccessible(GcsPathValidator.java:78) at org.apache.beam.sdk.util.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:62) at org.apache.beam.sdk.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:217) ... 15 more

解决方案

Somewhere in the documentation it was mentioned that I need to run the following .

gcloud auth application-default login

Obviously the exception does not help with figuring out this , I hope it helps someone else.

这篇关于Eclipse 上带有 Dataflow Runner 的 Apache Beam MinimalWordcount 示例的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！