如何在 Apache Beam 中一起使用 MapElements 和 KV? [英] How do I use MapElements and KV in together in Apache Beam?

问题描述

我想做这样的事情:

PCollection<String> a = whatever;
PCollection<KV<String, User>> b = a.apply(
        MapElements.into(TypeDescriptor.of(KV<String, User>.class))
        .via(s -> KV.of(s, new User(s))));

其中 User 是带有 Arvo 编码器和考虑字符串的构造函数的自定义数据类型.

Where User is a custom datatype with Arvo coder and a constructor that takes a string into account.

但是，我收到以下错误:

However, I get the following error:

无法从参数化类型中选择

Cannot select from parameterized type

我尝试将其更改为 TypeDescriptor.of(KV.class)，但是我得到:

I tried to change it to TypeDescriptor.of(KV.class) instead, but then I get:

不兼容的类型；必需的 PCollection> 但应用"被推断为 OutputT:不存在类型变量的实例，因此 PCollection 符合 PCollection>

Incompatible types; Required PCollection> but 'apply' was inferred to OutputT: no instance(s) of type variable(s) exists so that PCollection conforms to PCollection>

那么我应该如何将 KV 与 MapElements 一起使用?

So how am I suppose to use KV with MapElements?

我知道我想要做的是使用 ParDo 是可行的，我可以通过取消清除 new DoFn> 来明确指定如何进行类型擦除. 但 ParDo 不支持 lambda 函数.由于我们使用的是 Java 8，这似乎不太优雅......

I know that what I want to do is doable using ParDo where I could explicitly specify how to do Type Erasure by declearing new DoFn<String, KV<String, User>> but ParDo does not support lambda function. As we are using Java 8, this seems less elegant....

推荐答案

由于 输入擦除Java 在编译过程中，KV.class 被转化为 KV.class 而在运行时 KV.class 不是没有足够的信息来推断编码器，因为类型变量已被删除.

Due to type erasure in Java during compilation, KV<String, User>.class is transformed into KV.class and at runtime KV.class isn't enough information to infer a coder since the type variables have been erased.

要解决此限制，您需要使用一种机制来在编译后保留类型信息.例如，您可以使用:

To get around this limitation, you need to use a mechanism which preserves type information after compilation. For example you could use:

TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptor.of(User.class))

这与提供您自己的匿名类相同:

which is the same as providing your own anonymous class:

new TypeDescriptor<KV<String, User>> {}

提供绑定类型变量的匿名类是目前在 Java 中绕过类型擦除的方法之一.

Providing anonymous classes with type variables bound is one of the ways to get around type erasure in Java currently.

这篇关于如何在 Apache Beam 中一起使用 MapElements 和 KV?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！