Apache Beam:不可见的参数类型异常 [英] Apache Beam: Invisible parameter type exception
问题描述
我已经在Apache Beam中构建了一个小函数来执行查找/联接:给定一个将映射A映射到B,将另一个映射B映射到C,将返回的映射A映射回C.
I have built a small function in Apache Beam to perform a lookup/join: given a collection mapping A to B, and another collection mapping B to C, return a collection mapping A to C.
class Main {
private static <A,B,C> PCollection<KV<A,C>> lookup(
PCollection<KV<A,B>> collection,
PCollection<KV<B,C>> lookup
){
var leftTag = new TupleTag<A>();
var rightTag = new TupleTag<C>();
return KeyedPCollectionTuple.of(leftTag, collection.apply(KvSwap.create()))
.and(rightTag, lookup)
.apply(CoGroupByKey.create())
.apply(ParDo.of(new DoFn<KV<B, CoGbkResult>, KV<A, C>>() {
@ProcessElement
public void processElement(ProcessContext c) {
var value = c.element().getValue();
var right = value.getOnly(rightTag);
for (var i : value.getAll(leftTag)) {
c.output(KV.of(i, right));
}
}
}));
}
public static void main(String[] args) {
PipelineOptions options = PipelineOptionsFactory.create();
Pipeline p = Pipeline.create(options);
PCollection<KV<String, Long>> test = p.apply(Create.of(KV.of("a", 1L), KV.of("b", 2L)))
.setTypeDescriptor(TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.longs()));
PCollection<KV<Long, String>> test2 = p.apply(Create.of(KV.of(1L, "a"), KV.of(2L, "b")))
.setTypeDescriptor(TypeDescriptors.kvs(TypeDescriptors.longs(), TypeDescriptors.strings()));
var c = lookup(test, test2)
.setTypeDescriptor(TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.strings()));
p.run().waitUntilFinish();
}
}
不幸的是,在运行管道时出现以下错误:
Unfortunately, I get the following error when running the pipeline:
Exception in thread "main" org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException: Invisible parameter type of Main$1 arg0 for public Main$1$DoFnInvoker(Main$1)
at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
at org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
...
Caused by: java.lang.IllegalStateException: Invisible parameter type of Main$1 arg0 for public Main$1$DoFnInvoker(Main$1)
at org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.InstrumentedType$Default.validated(InstrumentedType.java:925)
at org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.MethodRegistry$Default.prepare(MethodRegistry.java:465)
at org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:170)
...
我似乎找不到太多有关此异常含义的文档.我认为这与类型擦除有关,并尝试使用TypeDescriptor随意添加代码,但没有成功.我该怎么办才能解决此错误?
I cant seem to find much documentation of what this exception means. I assumed this had something to do with type erasure and tried peppering the code liberally with TypeDescriptors, but with no success. What can I do to fix this error?
推荐答案
事实证明,这是由使用JDK 10引起的,Beam撰写本文时还不支持.切换到JDK 8解决了该问题.
Turns out that this was caused by using JDK 10, which is not supported by Beam as of writing. Switching to JDK 8 solved the problem.
这篇关于Apache Beam:不可见的参数类型异常的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!