如何在Apache Beam中一起使用MapElements和KV? [英] How do I use MapElements and KV in together in Apache Beam?

问题描述

我想做类似的事情:

PCollection<String> a = whatever;
PCollection<KV<String, User>> b = a.apply(
        MapElements.into(TypeDescriptor.of(KV<String, User>.class))
        .via(s -> KV.of(s, new User(s))));

其中用户"是具有Arvo编码器的自定义数据类型，并且是将字符串考虑在内的构造函数.

Where User is a custom datatype with Arvo coder and a constructor that takes a string into account.

但是，出现以下错误:

无法从参数化类型中选择

Cannot select from parameterized type

我尝试将其更改为TypeDescriptor.of(KV.class)，但随后得到:

I tried to change it to TypeDescriptor.of(KV.class) instead, but then I get:

不兼容的类型；必需的PCollection>，但是将"apply"推断为OutputT:不存在类型变量的实例，因此PCollection符合PCollection>

Incompatible types; Required PCollection> but 'apply' was inferred to OutputT: no instance(s) of type variable(s) exists so that PCollection conforms to PCollection>

那么我应该如何将KV与MapElements一起使用?

So how am I suppose to use KV with MapElements?

我知道我想做的事是可以使用ParDo进行的，在这里我可以通过清除new DoFn<String, KV<String, User>>来明确指定如何进行类型擦除，但是ParDo不支持lambda函数.当我们使用Java 8时，这似乎不太优雅....

I know that what I want to do is doable using ParDo where I could explicitly specify how to do Type Erasure by declearing new DoFn<String, KV<String, User>> but ParDo does not support lambda function. As we are using Java 8, this seems less elegant....

推荐答案

由于类型擦除Java 在编译过程中，KV<String, User>.class被转换为KV.class，并且在运行时KV.class信息不足以推断出编码器，因为类型变量已被删除.

Due to type erasure in Java during compilation, KV<String, User>.class is transformed into KV.class and at runtime KV.class isn't enough information to infer a coder since the type variables have been erased.

要解决此限制，您需要使用一种在编译后保留类型信息的机制.例如，您可以使用:

To get around this limitation, you need to use a mechanism which preserves type information after compilation. For example you could use:

TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptor.of(User.class))

与提供您自己的匿名类相同:

which is the same as providing your own anonymous class:

new TypeDescriptor<KV<String, User>> {}

提供绑定了类型变量的匿名类是当前在Java中解决类型擦除的方法之一.

Providing anonymous classes with type variables bound is one of the ways to get around type erasure in Java currently.

这篇关于如何在Apache Beam中一起使用MapElements和KV?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！