在apache beam中使用SpannerIO时出错 [英] Error using SpannerIO in apache beam
问题描述
此问题是对此问题的跟进。
我正在尝试使用apache beam从google spanner表中读取数据(然后进行一些数据处理)。我使用java SDK编写了以下最小示例:
This question is a follow-up to this one. I am trying to use apache beam to read data from a google spanner table (and then do some data processing). I wrote the following minimum example using the java SDK:
package com.google.cloud.dataflow.examples;
import java.io.IOException;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.io.gcp.spanner.SpannerIO;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.values.PCollection;
import com.google.cloud.spanner.Struct;
public class backup {
public static void main(String[] args) throws IOException {
PipelineOptions options = PipelineOptionsFactory.create();
Pipeline p = Pipeline.create(options);
PCollection<Struct> rows = p.apply(
SpannerIO.read()
.withInstanceId("my_instance")
.withDatabaseId("my_db")
.withQuery("SELECT t.table_name FROM information_schema.tables AS t")
);
PipelineResult result = p.run();
try {
result.waitUntilFinish();
} catch (Exception exc) {
result.cancel();
}
}
}
当我尝试执行使用DirectRunner的代码,我收到
以下错误消息:
When I try to execute the code using the DirectRunner, I get the following error message:
org.apache.beam.runners.direct.repackaged.com.google.common.util .concurrent.UncheckedExecutionException:
org.apache.beam.runners.direct.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.sdk.util.UserCodeException:
java.lang.NoClassDefFoundError:Could不初始化类
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
[...]引起:
org.apache .beam.sdk.util.UserCodeException:
java.lang.NoClassDefFoundError:无法初始化类
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
[...] Caused by: org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
[...]引起:
java.lang.NoClassDefFoundError:无法初始化类
com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
[...] Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor
或者,使用DataflowRunner:
Or, using the DataflowRunner:
org.apache.beam .runners.direct.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:
org.apache.beam.sdk.util.UserCodeException:
java.lang.NoSuchFieldError:
internal_static_google_rpc_LocalizedMessage_fieldAccessorTable
org.apache.beam.runners.direct.repackaged.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.beam.sdk.util.UserCodeException: java.lang.NoSuchFieldError: internal_static_google_rpc_LocalizedMessage_fieldAccessorTable
[...]引起:org.apache.beam.sdk.util.UserCodeException:
java.lang.NoSuchFieldError:
internal_static_google_rpc_LocalizedMessage_fieldAccessorTable
[...] Caused by: org.apache.beam.sdk.util.UserCodeException: java.lang.NoSuchFieldError: internal_static_google_rpc_LocalizedMessage_fieldAccessorTable
[...]引起:java.lang.NoSuchFieldError:
internal_static_google_rpc_LocalizedMessage_fieldAccessorTable
[...] Caused by: java.lang.NoSuchFieldError: internal_static_google_rpc_LocalizedMessage_fieldAccessorTable
在这两种情况下,错误信息都相当神秘,我找不到任何有关导致谷歌搜索错误的原因的明确想法。我也找不到使用SpannerIO模块的任何示例脚本。
In both cases, the error message is rather cryptic, and I could not find any clear ideas as to what causes the error from a google search. I also could not find any example scripts using the SpannerIO module.
这个错误是由于我的代码中的明显错误,还是由于安装错误导致的错误谷歌云工具?
Is this error due to an obvious error in my code, or is it due to a bad installation of the google cloud tools ?
推荐答案
此问题很可能是由此处描述的依赖项兼容性问题引起的: BEAM-2837 。以下是JIRA问题中的一条评论中描述的快速解决方法:
This issue is most likely caused by a dependency compatibility problem described here: BEAM-2837. Here's a quick workaround described in one of the comments in the JIRA issue:
<dependency>
<groupId>com.google.api.grpc</groupId>
<artifactId>grpc-google-common-protos</artifactId>
<version>0.1.9</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
<version>${beam.version}</version>
<exclusions>
<exclusion>
<groupId>com.google.api.grpc</groupId>
<artifactId>grpc-google-common-protos</artifactId>
</exclusion>
</exclusions>
</dependency>
明确定义所需的 com.google.api.grpc
依赖项并从 org.apache.beam
中排除该版本。
Explicitly define the required com.google.api.grpc
dependency and exclude the version from org.apache.beam
.
这篇关于在apache beam中使用SpannerIO时出错的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!