在 apache 光束中使用 SpannerIO 时出错 [英] Error using SpannerIO in apache beam

问题描述

这个问题是这个问题的后续.我正在尝试使用 apache beam 从谷歌扳手表中读取数据(然后进行一些数据处理).我使用 java SDK 编写了以下最小示例:

This question is a follow-up to this one. I am trying to use apache beam to read data from a google spanner table (and then do some data processing). I wrote the following minimum example using the java SDK:

package com.google.cloud.dataflow.examples;
import java.io.IOException;
import org.apache.beam.sdk.Pipeline;
import org.apache.beam.sdk.PipelineResult;
import org.apache.beam.sdk.io.gcp.spanner.SpannerIO;
import org.apache.beam.sdk.options.PipelineOptions;
import org.apache.beam.sdk.options.PipelineOptionsFactory;
import org.apache.beam.sdk.values.PCollection;
import com.google.cloud.spanner.Struct;

public class backup {

  public static void main(String[] args) throws IOException {
    PipelineOptions options = PipelineOptionsFactory.create();

    Pipeline p = Pipeline.create(options);
    PCollection<Struct> rows = p.apply(
            SpannerIO.read()
                .withInstanceId("my_instance")
                .withDatabaseId("my_db")
                .withQuery("SELECT t.table_name FROM information_schema.tables AS t")
                );
    
    PipelineResult result = p.run();
    try {
      result.waitUntilFinish();
    } catch (Exception exc) {
      result.cancel();
    }
  }
}

当我尝试使用 DirectRunner 执行代码时，我得到以下错误消息:

When I try to execute the code using the DirectRunner, I get the following error message:

org.apache.beam.runners.direct.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:

org.apache.beam.runners.direct.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:

org.apache.beam.sdk.util.UserCodeException:java.lang.NoClassDefFoundError:无法初始化类com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor

org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor

[...] 原因:org.apache.beam.sdk.util.UserCodeException:java.lang.NoClassDefFoundError:无法初始化类com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor

[...] Caused by: org.apache.beam.sdk.util.UserCodeException: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor

[...] 原因:java.lang.NoClassDefFoundError:无法初始化类com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor

[...] Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.google.cloud.spanner.spi.v1.SpannerErrorInterceptor

或者，使用 DataflowRunner:

Or, using the DataflowRunner:

org.apache.beam.runners.direct.repackaged.com.google.common.util.concurrent.UncheckedExecutionException:org.apache.beam.sdk.util.UserCodeException:java.lang.NoSuchFieldError:internal_static_google_rpc_LocalizedMessage_fieldAccessorTable

org.apache.beam.runners.direct.repackaged.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.beam.sdk.util.UserCodeException: java.lang.NoSuchFieldError: internal_static_google_rpc_LocalizedMessage_fieldAccessorTable

[...] 引起:org.apache.beam.sdk.util.UserCodeException:java.lang.NoSuchFieldError:internal_static_google_rpc_LocalizedMessage_fieldAccessorTable

[...] Caused by: org.apache.beam.sdk.util.UserCodeException: java.lang.NoSuchFieldError: internal_static_google_rpc_LocalizedMessage_fieldAccessorTable

[...] 引起:java.lang.NoSuchFieldError:internal_static_google_rpc_LocalizedMessage_fieldAccessorTable

[...] Caused by: java.lang.NoSuchFieldError: internal_static_google_rpc_LocalizedMessage_fieldAccessorTable

在这两种情况下，错误消息都相当神秘，我无法从谷歌搜索中找到任何关于导致错误的明确想法.我也找不到任何使用 SpannerIO 模块的示例脚本.

In both cases, the error message is rather cryptic, and I could not find any clear ideas as to what causes the error from a google search. I also could not find any example scripts using the SpannerIO module.

这个错误是由于我的代码中的明显错误引起的，还是由于谷歌云工具安装不当引起的?

Is this error due to an obvious error in my code, or is it due to a bad installation of the google cloud tools ?

推荐答案

此问题很可能是由此处描述的依赖项兼容性问题引起的:BEAM-2837.以下是 JIRA 问题中的一条评论中描述的快速解决方法:

This issue is most likely caused by a dependency compatibility problem described here: BEAM-2837. Here's a quick workaround described in one of the comments in the JIRA issue:

<dependency>
    <groupId>com.google.api.grpc</groupId>
    <artifactId>grpc-google-common-protos</artifactId>
    <version>0.1.9</version>
</dependency>

<dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-sdks-java-io-google-cloud-platform</artifactId>
    <version>${beam.version}</version>
    <exclusions>
        <exclusion>
            <groupId>com.google.api.grpc</groupId>
            <artifactId>grpc-google-common-protos</artifactId>
        </exclusion>
    </exclusions>
</dependency>

明确定义所需的 com.google.api.grpc 依赖项并从 org.apache.beam 中排除版本.

Explicitly define the required com.google.api.grpc dependency and exclude the version from org.apache.beam.

这篇关于在 apache 光束中使用 SpannerIO 时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！