Apache Beam:无法找到gs的注册商 [英] Apache Beam: Unable to find registrar for gs
问题描述
Beam is using both Google's auto/value and auto/service tools.
我想运行带有Dataflow流水线的管道,数据存储在Google云端存储上。
I want to run a pipeline with Dataflow runner and data is stored on Google Cloud Storage.
我' ve添加了依赖项:
I've added a dependencies:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.0.0</version>
</dependency>
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-sdks-java-extensions-google-cloud-platform-core</artifactId>
<version>2.0.0</version>
</dependency>
我可以从IntelliJ启动管道。但是当jar通过 mvn包
编译并使用 java -jar
运行时,它会抛出一个错误:
I'm able to start the pipeline from the IntelliJ. But when the jar is compiled through a mvn package
and run with java -jar
it throws an error:
java.lang.IllegalStateException: Unable to find registrar for gs
fatjar包含 maven-assembly-plugin
。 GcsFileSystemRegistrar
类在jar中。
The fatjar is package with maven-assembly-plugin
. GcsFileSystemRegistrar
class is in the jar.
推荐答案
问题在于你正在建立你的脂肪的方式。 maven-assembly-plugin
未正确处理与 ServiceLoader
关联的文件。 ServiceLoader
依赖中列出的条目META-INF / services / org.apache.beam.sdk.io.FileSystemRegistrar
对于每个实现,以便Java知道如何找到它们。
The issue is in the way that you are building your fatjar. The maven-assembly-plugin
is not handling files associated with ServiceLoader
correctly. ServiceLoader
relies on entries being listed within META-INF/services/org.apache.beam.sdk.io.FileSystemRegistrar
for each implementation so that Java knows how to find them.
META-INF / services / org.apache.beam.sdk的内容你的fatjar中的.io.FileSystemRegistrar
可能只有:
org.apache.beam.sdk.io.LocalFileSystemRegistrar
你需要列出它(以及你想要的任何其他实现):
You need to have it list (and any other implementations that you want):
org.apache.beam.sdk.io.LocalFileSystemRegistrar
org.apache.beam.sdk.extensions.gcp.storage.GcsFileSystemRegistrar
您最好的选择是使用了解这些 ServiceLoader的工具
配置为使用 maven-shade-plugin 等要求-shade-plugin / examples / resource-transformers.html#ServicesResourceTransformerrel =no referrer> ServicesResourceTransformer 来构建你的fatjar。
Your best bet is to use a tool which understands these ServiceLoader
requirements like the maven-shade-plugin
when configured to use the ServicesResourceTransformer to build your fatjar.
这篇关于Apache Beam:无法找到gs的注册商的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!