如何在Java中的Apache Spark中将DataFrame转换为Dataset? [英] How to convert DataFrame to Dataset in Apache Spark in Java?
本文介绍了如何在Java中的Apache Spark中将DataFrame转换为Dataset?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我可以很容易地在Scala中将DataFrame转换为Dataset:
I can convert DataFrame to Dataset in Scala very easy:
case class Person(name:String, age:Long)
val df = ctx.read.json("/tmp/persons.json")
val ds = df.as[Person]
ds.printSchema
但是在Java版本中,我不知道如何将Dataframe转换为Dataset?有想法吗?
but in Java version I don't know how to convert Dataframe to Dataset? Any Idea?
我的努力是:
DataFrame df = ctx.read().json(logFile);
Encoder<Person> encoder = new Encoder<>();
Dataset<Person> ds = new Dataset<Person>(ctx,df.logicalPlan(),encoder);
ds.printSchema();
但是编译器会说:
Error:(23, 27) java: org.apache.spark.sql.Encoder is abstract; cannot be instantiated
已修改(解决方案):
基于@ Leet-Falcon答案的解决方案:
Edited(Solution):
solution based on @Leet-Falcon answers:
DataFrame df = ctx.read().json(logFile);
Encoder<Person> encoder = Encoders.bean(Person.class);
Dataset<Person> ds = new Dataset<Person>(ctx, df.logicalPlan(), encoder);
推荐答案
通过在
List<String> data = Arrays.asList("abc", "abc", "xyz");
Dataset<String> ds = context.createDataset(data, Encoders.STRING());
编码器可以组成元组:
Encoder<Tuple2<Integer, String>> encoder2 = Encoders.tuple(Encoders.INT(), Encoders.STRING());
List<Tuple2<Integer, String>> data2 = Arrays.asList(new scala.Tuple2(1, "a");
Dataset<Tuple2<Integer, String>> ds2 = context.createDataset(data2, encoder2);
或通过 查看全文