如何在 Java 的 Apache Spark 中将 DataFrame 转换为 Dataset? [英] How to convert DataFrame to Dataset in Apache Spark in Java?

问题描述

我可以很容易地在 Scala 中将 DataFrame 转换为 Dataset:

I can convert DataFrame to Dataset in Scala very easy:

case class Person(name:String, age:Long)
val df = ctx.read.json("/tmp/persons.json")
val ds = df.as[Person]
ds.printSchema

但是在 Java 版本中我不知道如何将 Dataframe 转换为 Dataset?有什么想法吗?

but in Java version I don't know how to convert Dataframe to Dataset? Any Idea?

我的努力是:

DataFrame df = ctx.read().json(logFile);
Encoder<Person> encoder = new Encoder<>();
Dataset<Person> ds = new Dataset<Person>(ctx,df.logicalPlan(),encoder);
ds.printSchema();

但编译器说:

Error:(23, 27) java: org.apache.spark.sql.Encoder is abstract; cannot be instantiated

已编辑(解决方案):

基于@Leet-Falcon 答案的解决方案:

Edited(Solution):

solution based on @Leet-Falcon answers:

DataFrame df = ctx.read().json(logFile);
Encoder<Person> encoder = Encoders.bean(Person.class);
Dataset<Person> ds = new Dataset<Person>(ctx, df.logicalPlan(), encoder);

推荐答案

Spark 官方文档建议在 Dataset API 如下:

Official Spark docs suggest in Dataset API the following:

Java 编码器通过在 编码器.

Java Encoders are specified by calling static methods on Encoders.

List<String> data = Arrays.asList("abc", "abc", "xyz");
Dataset<String> ds = context.createDataset(data, Encoders.STRING());

编码器可以组合成元组:

Encoders can be composed into tuples:

Encoder<Tuple2<Integer, String>> encoder2 = Encoders.tuple(Encoders.INT(), Encoders.STRING());
List<Tuple2<Integer, String>> data2 = Arrays.asList(new scala.Tuple2(1, "a");
Dataset<Tuple2<Integer, String>> ds2 = context.createDataset(data2, encoder2);

或者通过 编码器#bean:

Encoders.bean(MyClass.class);

这篇关于如何在 Java 的 Apache Spark 中将 DataFrame 转换为 Dataset?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！