从案例类生成 Spark StructType/Schema [英] Generate a Spark StructType / Schema from a case class
问题描述
如果我想从 case class
中创建一个 StructType
(即一个 DataFrame.schema
),有没有办法做到它没有创建DataFrame
?我可以轻松做到:
If I wanted to create a StructType
(i.e. a DataFrame.schema
) out of a case class
, is there a way to do it without creating a DataFrame
? I can easily do:
case class TestCase(id: Long)
val schema = Seq[TestCase]().toDF.schema
但是当我想要的只是架构时,实际创建一个 DataFrame
似乎有点过分.
But it seems overkill to actually create a DataFrame
when all I want is the schema.
(如果你很好奇,这个问题背后的原因是我定义了一个 UserDefinedAggregateFunction
,为此你覆盖了几个返回 StructTypes
的方法和我使用案例类.)
(If you are curious, the reason behind the question is that I am defining a UserDefinedAggregateFunction
, and to do so you override a couple of methods that return StructTypes
and I use case classes.)
推荐答案
你也可以这样做 SQLContext.createDataFrame
做到了:
You can do it the same way SQLContext.createDataFrame
does it:
import org.apache.spark.sql.catalyst.ScalaReflection
val schema = ScalaReflection.schemaFor[TestCase].dataType.asInstanceOf[StructType]
这篇关于从案例类生成 Spark StructType/Schema的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!