Spark Dataset中的SortedMap不可序列化错误 [英] SortedMap non serializable error in Spark Dataset

问题描述

似乎scala.collection.SortedMap无法序列化?

It seems like scala.collection.SortedMap is not serializable?

简单的代码示例:

case class MyClass(s: scala.collection.SortedMap[String, String] = SortedMap[String, String]())

object MyClass {
  def apply(i: Int): MyClass = MyClass()
}

import sparkSession.implicits._

List(MyClass(1), MyClass()).toDS().show(2)

会返回:

+-----+
|    s|
+-----+
|Map()|
|Map()|
+-----+

另一方面，take()将在执行时严重失败:

On the other hand, take() will fail miserably at execution time:

List(MyClass(1), MyClass()).toDS().take(2)

错误codegen.CodeGenerator:编译失败:org.codehaus.commons.compiler.CompileException:文件'Generated.java'，第116行，第100列:找不到适用于实际参数"scala.collection.Map的适用构造函数/方法;候选对象是:"com.caspida.algorithms.security.offline.ex渗透threat.MyClass(s​​cala.collection.SortedMap)"

ERROR codegen.CodeGenerator: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 116, Column 100: No applicable constructor/method found for actual parameters "scala.collection.Map"; candidates are: "com.caspida.algorithms.security.offline.exfiltrationthreat.MyClass(scala.collection.SortedMap)"

推荐答案

Spark支持的Scala类型(从2.1.0版本开始)不包括scala.collection.SortedMap).可以在此处找到受支持的类型的列表:

The supported Scala types for Spark (as of 2.1.0) do not include scala.collection.SortedMap). A list of supported types can be found here:

https://spark.apache.org/docs/latest/sql-programming-guide.html#data-types

如链接所建议，地图的受支持类型为 scala.collection.Map ，因此可进行以下操作:

As the link suggest, the supported type for Maps is scala.collection.Map so the following works:

case class MyClass(s: scala.collection.Map[String, String] = SortedMap[String, String]())


scala> spark.createDataset( MyClass() :: Nil ).collect()
res: Array[MyClass2] = Array(MyClass(Map()))

这篇关于Spark Dataset中的SortedMap不可序列化错误的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！