将Spark DataFrame转换为Pojo对象 [英] Convert Spark DataFrame to Pojo Object
问题描述
请参见以下代码:
//Create Spark Context
SparkConf sparkConf = new SparkConf().setAppName("TestWithObjects").setMaster("local");
JavaSparkContext javaSparkContext = new JavaSparkContext(sparkConf);
//Creating RDD
JavaRDD<Person> personsRDD = javaSparkContext.parallelize(persons);
//Creating SQL context
SQLContext sQLContext = new SQLContext(javaSparkContext);
DataFrame personDataFrame = sQLContext.createDataFrame(personsRDD, Person.class);
personDataFrame.show();
personDataFrame.printSchema();
personDataFrame.select("name").show();
personDataFrame.registerTempTable("peoples");
DataFrame result = sQLContext.sql("SELECT * FROM peoples WHERE name='test'");
result.show();
此后,我需要将DataFrame-'result'转换为Person Object或List.预先感谢.
After this I need to convert the DataFrame - 'result' to Person Object or List. Thanks in advance.
推荐答案
DataFrame只是Dataset [Row]的类型别名.与强类型的Scala/Java数据集附带的类型转换"相反,这些操作也称为非类型转换".
DataFrame is simply a type alias of Dataset[Row] . These operations are also referred as "untyped transformations" in contrast to "typed transformations" that come with strongly typed Scala/Java Datasets.
从数据集[行]到数据集[人]的转换非常简单
The conversion from Dataset[Row] to Dataset[Person] is very simple in spark
DataFrame result = sQLContext.sql("SELECT * FROM peoples WHERE name='test'");
这时,Spark会将您的数据转换为DataFrame = Dataset [Row],这是通用Row对象的集合,因为它不知道确切的类型.
At this point, Spark converts your data into DataFrame = Dataset[Row], a collection of generic Row object, since it does not know the exact type.
// Create an Encoders for Java beans
Encoder<Person> personEncoder = Encoders.bean(Person.class);
Dataset<Person> personDF = result.as(personEncoder);
personDF.show();
现在,Spark会按照类Person的指示转换Dataset [Row]-> Dataset [Person]类型特定的Scala/Java JVM对象.
Now, Spark converts the Dataset[Row] -> Dataset[Person] type-specific Scala / Java JVM object, as dictated by the class Person.
请参阅数据砖提供的以下链接以获取更多详细信息
Please refer to below link provided by databricks for further details
这篇关于将Spark DataFrame转换为Pojo对象的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!