在Eclipse上使用Dataflow Runner的Apache Beam MinimalWordcount示例 [英] Apache Beam MinimalWordcount example with Dataflow Runner on eclipse

问题描述

我试图在Windows上使用MinimalWordCount在Windows上使用Eclipse的DataFlowRunner运行MinimalWordCount示例->使用eclipse中的With As Java Application运行，其示例代码与使用gcs存储区的示例相同，但是我始终如一地得到除了例外，有人可以让我知道这里的问题吗?

I am trying to run the MinimalWordCount example using the DataFlowRunner from Eclipse on windows using MinimalWordCount -->Run As Java Application from with in eclipse , its the same stock code from the example using my gcs bucket , however I consistently get the following exception , can some one let me know whats the issue here?

1. 我已验证存储桶名称正确.
2. 我已经在Windows计算机上运行了gcloud init.

线程"main"中的异常java.lang.RuntimeException:无法执行 从工厂方法构造实例 DataflowRunner#fromOptions(接口 org.apache.beam.sdk.options.PipelineOptions) 在org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233) 在org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162) 在org.apache.beam.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:56) 在org.apache.beam.sdk.Pipeline.create(Pipeline.java:135) 在com.dynaobject.apachebeamexperiment.MinimalWordCount.main(MinimalWordCount.java:77)处 造成原因:java.lang.reflect.InvocationTargetException 在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)处 在sun.reflect.NativeMethodAccessorImpl.invoke(未知来源) 在sun.reflect.DelegatingMethodAccessorImpl.invoke(未知来源) 在java.lang.reflect.Method.invoke(未知来源) 在org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222) ...另外4个 原因:java.lang.IllegalArgumentException:DataflowRunner需要gcpTempLocation，但无法从中检索值 管道选项 在org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:212) ...另外9个 造成原因:java.lang.IllegalArgumentException:为gcpTempLocation构造默认值时出错:tempLocation不是 有效的GCS路径，gs://tempxyz. 在org.apache.beam.sdk.options.GcpOptions $ GcpTempLocationFactory.create(GcpOptions.java:219) 在org.apache.beam.sdk.options.GcpOptions $ GcpTempLocationFactory.create(GcpOptions.java:205) 在org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:575) 在org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:516) 在org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:155) 在com.sun.proxy.$ Proxy15.getGcpTempLocation(未知来源) 在org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:210) ...另外9个 原因:java.lang.IllegalArgumentException:输出路径不存在或不可写:gs://tempxyz 在org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:191) 在org.apache.beam.sdk.util.GcsPathValidator.verifyPathIsAccessible(GcsPathValidator.java:78) 在org.apache.beam.sdk.util.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:62) 在org.apache.beam.sdk.options.GcpOptions $ GcpTempLocationFactory.create(GcpOptions.java:217) ...另外15个

Exception in thread "main" java.lang.RuntimeException: Failed to construct instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:233) at org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162) at org.apache.beam.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:56) at org.apache.beam.sdk.Pipeline.create(Pipeline.java:135) at com.dynaobject.apachebeamexperiment.MinimalWordCount.main(MinimalWordCount.java:77) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222) ... 4 more Caused by: java.lang.IllegalArgumentException: DataflowRunner requires gcpTempLocation, but failed to retrieve a value from PipelineOptions at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:212) ... 9 more Caused by: java.lang.IllegalArgumentException: Error constructing default value for gcpTempLocation: tempLocation is not a valid GCS path, gs://tempxyz. at org.apache.beam.sdk.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:219) at org.apache.beam.sdk.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:205) at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:575) at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:516) at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:155) at com.sun.proxy.$Proxy15.getGcpTempLocation(Unknown Source) at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:210) ... 9 more Caused by: java.lang.IllegalArgumentException: Output path does not exist or is not writeable: gs://tempxyz at org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:191) at org.apache.beam.sdk.util.GcsPathValidator.verifyPathIsAccessible(GcsPathValidator.java:78) at org.apache.beam.sdk.util.GcsPathValidator.validateOutputFilePrefixSupported(GcsPathValidator.java:62) at org.apache.beam.sdk.options.GcpOptions$GcpTempLocationFactory.create(GcpOptions.java:217) ... 15 more

推荐答案

在文档中的某些地方提到了我需要运行以下内容.

Somewhere in the documentation it was mentioned that I need to run the following .

gcloud auth应用程序-默认登录名

gcloud auth application-default login

显然，该异常无助于解决这一问题，希望对其他人有所帮助.

Obviously the exception does not help with figuring out this , I hope it helps someone else.

这篇关于在Eclipse上使用Dataflow Runner的Apache Beam MinimalWordcount示例的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！