Spark Scala:检索架构并将其存储 [英] Spark Scala: retrieve the schema and store it

问题描述

是否可以检索RDD的架构并将其存储在变量中?因为我想使用相同的架构从另一个RDD创建一个新的数据框.例如，以下是我希望拥有的东西:

Is it possible to retrieve the schema of an RDD and store it in a variable? Because I want to create a new data frame from another RDD using the same schema. For example, below is what I am hoping to have:

val schema = oldDF.getSchema()
val newDF = sqlContext.createDataFrame(rowRDD, schema)

假设我已经具有RDD[org.apache.spark.sql.Row]格式的rowRDD，这可能吗?

Assuming I already have rowRDD in the format of RDD[org.apache.spark.sql.Row] , is this something possible?

推荐答案

只需使用schema属性

val oldDF = sqlContext.createDataFrame(sc.parallelize(Seq(("a", 1))))
val rowRDD = sc.parallelize(Seq(Row("b", 2))

sqlContext.createDataFrame(rowRDD, oldDF.schema)

这篇关于Spark Scala:检索架构并将其存储的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！