斯卡拉火花机壳类的普通编码器 [英] scala generic encoder for spark case class
本文介绍了斯卡拉火花机壳类的普通编码器的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我怎样才能得到这个方法来编译。奇怪的是,隐含的火花已经被导入。
def loadDsFromHive [T<:Product](tableName:String,spark:SparkSession) :数据集[T] = {
导入spark.implicits._
spark.sql(sSELECT * FROM $ tableName)。as [T]
}
这是错误:
无法找到存储在数据集中的类型的编码器。通过导入spark.implicits._支持原始类型(Int,String等)和Product类型(case类)。将来的发行版中将添加对序列化其他类型的支持。
[error] spark.sql(sSELECT * FROM $ tableName)。as [T]
org.apache.spark.sql.SQLImplicits 的源代码,你需要类型类
TypeTag
Encoder
存在: import scala.reflect.runtime.universe.TypeTag
def loadDsFromHive [T<:Product:TypeTag](tableName:String,spark:SparkSession):Dataset [T ] = ...
How can I get this method to compile. Strangely, sparks implicit are already imported.
def loadDsFromHive[T <: Product](tableName: String, spark: SparkSession): Dataset[T] = {
import spark.implicits._
spark.sql(s"SELECT * FROM $tableName").as[T]
}
This is the error:
Unable to find encoder for type stored in a Dataset. Primitive types (Int, String, etc) and Product types (case classes) are supported by importing spark.implicits._ Support for serializing other types will be added in future releases.
[error] spark.sql(s"SELECT * FROM $tableName").as[T]
解决方案
According to the source code for org.apache.spark.sql.SQLImplicits
, you need the type class TypeTag
for your type, in order for the implicit Encoder
to exist:
import scala.reflect.runtime.universe.TypeTag
def loadDsFromHive[T <: Product: TypeTag](tableName: String, spark: SparkSession): Dataset[T] = ...
这篇关于斯卡拉火花机壳类的普通编码器的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文