将ADT /密封性状层次结构编码为Spark DataSet列 [英] Encode an ADT / sealed trait hierarchy into Spark DataSet column
问题描述
如果我想在Spark DataSet 列,什么是最好的编码策略?
例如,如果我有一个ADT,叶类型存储不同类型的数据:
sealed trait职业
案例对象软件工程师扩展职业
案例类向导(级别:Int)扩展职业
案例类其他(description:String)extends职业
最好的方法是构建一个:
org.apache.spark.sql。 DataSet [职业]
/ strong>现在没有很好的解决方案,并且给出了Spark SQL / Dataset
实现,在可预见的将来不太可能有一个。
您可以使用通用的 kryo
或 java
编码器
val职业:Seq [职业] = Seq(SoftwareEngineer,向导(1),其他(foo))
spark.createDataset(职业)(org.apache。 spark.sql.Encoders.kryo [职业])
但在实践中几乎没有用。 >
UDT API现在提供了另一种可能的方法(Spark 1.6
, 2.0
, 2.1-SNAPSHOT
)它是私有的,需要相当多的样板代码(您可以检查 oasml.linalg.VectorUDT
查看示例实现)。
If I want to store an Algebraic Data Type (ADT) (ie a Scala sealed trait hierarchy) within a Spark DataSet column, what is the best encoding strategy?
For example, if I have an ADT where the leaf types store different kinds of data:
sealed trait Occupation
case object SoftwareEngineer extends Occupation
case class Wizard(level: Int) extends Occupation
case class Other(description: String) extends Occupation
Whats the best way to construct a:
org.apache.spark.sql.DataSet[Occupation]
TL;DR There is no good solution right now, and given Spark SQL / Dataset
implementation, it is unlikely there will be one in the foreseeable future.
You can use generic kryo
or java
encoder
val occupation: Seq[Occupation] = Seq(SoftwareEngineer, Wizard(1), Other("foo"))
spark.createDataset(occupation)(org.apache.spark.sql.Encoders.kryo[Occupation])
but is hardly useful in practice.
UDT API provides another possible approach as for now (Spark 1.6
, 2.0
, 2.1-SNAPSHOT
) it is private and requires quite a lot boilerplate code (you can check o.a.s.ml.linalg.VectorUDT
to see example implementation).
这篇关于将ADT /密封性状层次结构编码为Spark DataSet列的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!