Apache Beam:无法找到 gs 的注册商 [英] Apache Beam: Unable to find registrar for gs

问题描述

Beam 同时使用 Google 的 auto/value 和 auto/service 工具.

Beam is using both Google's auto/value and auto/service tools.

我想使用 Dataflow 运行器运行管道，数据存储在 Google Cloud Storage 上.

I want to run a pipeline with Dataflow runner and data is stored on Google Cloud Storage.

我添加了一个依赖项:

<dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
    <version>2.0.0</version>
</dependency>

<dependency>
    <groupId>org.apache.beam</groupId>
    <artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
    <version>2.0.0</version>
</dependency>

我可以从 IntelliJ 启动管道.但是当 jar 通过 mvn package 编译并使用 java -jar 运行时，它会抛出错误:

I'm able to start the pipeline from the IntelliJ. But when the jar is compiled through a mvn package and run with java -jar it throws an error:

java.lang.IllegalStateException: Unable to find registrar for gs

fatjar 是带有 maven-assembly-plugin 的包.GcsFileSystemRegistrar 类在 jar 中.

The fatjar is package with maven-assembly-plugin. GcsFileSystemRegistrar class is in the jar.

推荐答案

问题在于您构建 fatjar 的方式.maven-assembly-plugin 未正确处理与 ServiceLoader 关联的文件.ServiceLoader 依赖于每个实现的 META-INF/services/org.apache.beam.sdk.io.FileSystemRegistrar 中列出的条目，以便 Java 知道如何找到它们.

The issue is in the way that you are building your fatjar. The maven-assembly-plugin is not handling files associated with ServiceLoader correctly. ServiceLoader relies on entries being listed within META-INF/services/org.apache.beam.sdk.io.FileSystemRegistrar for each implementation so that Java knows how to find them.

fatjar 中 META-INF/services/org.apache.beam.sdk.io.FileSystemRegistrar 的内容可能只有:

The contents of the META-INF/services/org.apache.beam.sdk.io.FileSystemRegistrar in your fatjar is likely only:

org.apache.beam.sdk.io.LocalFileSystemRegistrar

您需要列出它(以及您想要的任何其他实现):

You need to have it list (and any other implementations that you want):

org.apache.beam.sdk.io.LocalFileSystemRegistrar
org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar

当配置为使用 ServicesResourceTransformer 来构建你的 fatjar.

Your best bet is to use a tool which understands these ServiceLoader requirements like the maven-shade-plugin when configured to use the ServicesResourceTransformer to build your fatjar.

这篇关于Apache Beam:无法找到 gs 的注册商的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！