Apache Beam 和 BigQuery [英] Apache beam and BigQuery

问题描述

我正在尝试执行 apache beam sdk 2.4 和库 com.google.cloud.bigquery 但出现异常

线程主java.lang.NoSuchMethodError com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(LjavalangString;)LcomgoogleapiclientgoogleapisservicesAbstractGoogleClient$Builder中的异常；在 com.google.api.services.bigquery.Bigquery$Builder.setBatchPath(Bigquery.java3519)导入 com.google.cloud.bigquery.*;<依赖><groupId>com.google.cloud.dataflow</groupId><artifactId>google-cloud-dataflow-java-sdk-all</artifactId><version>[2.4.0, 2.99)</version></依赖><!-- slf4j API 前端与 JUL 后端的绑定--><依赖><groupId>org.slf4j</groupId><artifactId>slf4j-api</artifactId><相反在>1.7.14</version></依赖><依赖><groupId>org.slf4j</groupId><artifactId>slf4j-jdk14</artifactId><version>1.7.14</version></依赖><依赖><groupId>com.google.cloud</groupId><artifactId>google-cloud-bigquery</artifactId><version>1.35.0</version></依赖></依赖项>

<块引用>

 apache beam 2.4 使用的代码示例

import com.google.api.services.bigquery.model.TableFieldSchema;导入 com.google.api.services.bigquery.model.TableRow;导入 com.google.api.services.bigquery.model.TableSchema;

<块引用>

如果我使用库 om.google.api.services.bigquery.编码示例运行成功.

它内部使用的是什么库?

我应该使用什么库?com.google.api.services.bigquery 还是 com.google.cloud.bigquery?

解决方案

Beam 2.4 正在使用 com.google.api.services.bigquery v2-rev374-1.22.0.这也是您应该与 Beam 2.4 一起使用的库.com.google.api.services.bigquery 库已过时，并且有一个 open问题来升级它.遗憾的是，Beam 中的其他组件依赖于与 com.google.cloud.bigquery 不兼容的库，并且需要先升级这些库才能使用新库.

如果你还没有这样做，你应该看看BigQuery IO，它为 Beam 的常见 BigQuery 操作提供了封装.

I'm trying to execute apache beam sdk 2.4 and the libraries com.google.cloud.bigquery but thow exception

Exception in thread main java.lang.NoSuchMethodError com.google.api.client.googleapis.services.json.AbstractGoogleJsonClient$Builder.setBatchPath(LjavalangString;)LcomgoogleapiclientgoogleapisservicesAbstractGoogleClient$Builder;
at com.google.api.services.bigquery.Bigquery$Builder.setBatchPath(Bigquery.java3519)

import com.google.cloud.bigquery.*;

<dependency>
    <groupId>com.google.cloud.dataflow</groupId>
    <artifactId>google-cloud-dataflow-java-sdk-all</artifactId>
    <version>[2.4.0, 2.99)</version>
</dependency>

<!-- slf4j API frontend binding with JUL backend -->
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <versi

on>1.7.14</version>
    </dependency>

   <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-jdk14</artifactId>
        <version>1.7.14</version>
    </dependency>

    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-bigquery</artifactId>
        <version>1.35.0</version>
    </dependency>

</dependencies>

the code example apache beam 2.4 using

import com.google.api.services.bigquery.model.TableFieldSchema;
import com.google.api.services.bigquery.model.TableRow;
import com.google.api.services.bigquery.model.TableSchema;

If I use the the libraries om.google.api.services.bigquery. the code example run successfully .

What library is it using internally?

What library should I use? com.google.api.services.bigquery or com.google.cloud.bigquery?

解决方案

Beam 2.4 is using com.google.api.services.bigquery v2-rev374-1.22.0. This is also the library that you should use with Beam 2.4. The com.google.api.services.bigquery library is out of date and there is an open issue to upgrade it. Unfortunately other components in Beam depend on libraries which are incompatible with com.google.cloud.bigquery, and those need to be upgraded before the new library can be used.

If you aren't already doing so, you should take a look at the BigQuery IO, which provides a wrapper to common BigQuery operations for Beam.

这篇关于Apache Beam 和 BigQuery的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！