Beam:无法序列化和反序列化属性“awsCredentialsProvider" [英] Beam: Failed to serialize and deserialize property 'awsCredentialsProvider
问题描述
我一直在使用 Beam 管道 examples 作为指南,试图从 S3 为我的管道加载文件.就像在示例中一样,我定义了自己的 PipelineOptions
,它也扩展了 S3Options 并且我正在尝试使用 DefaultAWSCredentialsProviderChain.配置它的代码是:
I have been using a Beam pipeline examples as a guide in an attempt to load files from S3 for my pipeline. Like in the examples I have defined my own PipelineOptions
that also extends S3Options and I am attempting to use the DefaultAWSCredentialsProviderChain. The code to configure this is:
MyPipelineOptions options = PipelineOptionsFactory.fromArgs(args).as(MyPipelineOptions.class);
options.setAwsCredentialsProvider(new DefaultAWSCredentialsProviderChain());
options.setAwsRegion("us-east-1");
runPipeline(options);
当我从 Intellij 运行它时,它使用 Direct Runner 运行良好但是当我将它打包为 jar 并执行它(也使用 Direct Runner)时,我看到:
When I run it from Intellij it works fine using the Direct Runner but when I package it as a jar and it execute it (also using the Direct Runner) I see:
Exception in thread "main" java.lang.IllegalArgumentException: PipelineOptions specified failed to serialize to JSON.
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:166)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
at a.b.c.beam.CleanSkeleton.runPipeline(CleanSkeleton.java:69)
at a.b.c.beam.CleanSkeleton.main(CleanSkeleton.java:53)
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Unexpected IOException (of type java.io.IOException): Failed to serialize and deserialize property 'awsCredentialsProvider' with value 'com.amazonaws.auth.DefaultAWSCredentialsProviderChain@40f33492'
at com.fasterxml.jackson.databind.JsonMappingException.fromUnexpectedIOE(JsonMappingException.java:338)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsBytes(ObjectMapper.java:3247)
at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:163)
... 5 more
我正在使用 gradle 构建我的 jar 并完成以下任务:
I am using gradle to build my jar with the following task:
jar {
manifest {
attributes (
'Main-Class': 'a.b.c.beam.CleanSkeleton'
)
}
from {
configurations.runtimeClasspath.collect { it.isDirectory() ? it : zipTree(it) }
}
from('src') {
include '/main/resources/*'
}
zip64 true
exclude 'META-INF/*.RSA', 'META-INF/*.SF', 'META-INF/*.DSA'
}
推荐答案
出现问题是因为在创建 fat/uber jar 时,META-INF/serivces
中的文件被覆盖通过重复文件.特别是 com.fasterxml.jackson.databind.Module
需要定义许多 Jackson 模块但缺少的地方.其中包括 org.apache.beam.sdk.io.aws.options.AwsModule
和 com.fasterxml.jackson.datatype.joda.JodaModule
.DirectRunner
中的代码像这样实例化 ObjectMapper
:
The problem was occuring because when the the fat/uber jar was being created, files in META-INF/serivces
where being overwritten by duplicate files. Specifically com.fasterxml.jackson.databind.Module
where a number of Jackson modules needed to be defined but where missing. These include org.apache.beam.sdk.io.aws.options.AwsModule
and com.fasterxml.jackson.datatype.joda.JodaModule
. The code in the DirectRunner
instantiates the ObjectMapper
like so :
new ObjectMapper()
.registerModules(ObjectMapper.findModules(ReflectHelpers.findClassLoader()));
ObjectMapper::findModules
依赖于 java.util.ServiceLoader
,它从 META-INF/services/
文件中定位服务.
ObjectMapper::findModules
relies on java.util.ServiceLoader
which locates services from META-INF/services/
files.
解决方案是使用 gradle Shadow plugin 来构建 fat/uber jar 并对其进行配置合并服务文件:
The solution was to use the gradle Shadow plugin to build the fat/uber jar and configure it to merge the services files:
apply plugin: 'com.github.johnrengelman.shadow'
shadowJar {
mergeServiceFiles()
zip64 true
}
这篇关于Beam:无法序列化和反序列化属性“awsCredentialsProvider"的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!