不支持 Any 类型的架构 [英] Schema for type Any is not supported

问题描述

我正在尝试创建一个 spark UDF，以从用户定义的案例类中提取(键、值)对的映射.

I'm trying to create a spark UDF to extract a Map of (key, value) pairs from a User defined case class.

scala 函数似乎工作正常，但是当我尝试在 spark2.0 中将其转换为 UDF 时，我遇到了不支持任何类型的架构"错误.

The scala function seems to work fine, but when I try to convert that to a UDF in spark2.0, I'm running into the " Schema for type Any is not supported" error.

case class myType(c1: String, c2: Int)
def getCaseClassParams(cc: Product): Map[String, Any] = {

    cc
      .getClass
      .getDeclaredFields // all field names
      .map(_.getName)
      .zip(cc.productIterator.to) // zipped with all values
      .toMap

  }

但是当我尝试将函数值实例化为 UDF 时，会导致以下错误 -

But when I try to instantiate a function value as a UDF it results in the following error -

val ccUDF = udf{(cc: Product, i: String) => getCaseClassParams(cc).get(i)}

java.lang.UnsupportedOperationException: Schema for type Any is not supported
  at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:716)
  at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:668)
  at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:654)
  at org.apache.spark.sql.functions$.udf(functions.scala:2841)

推荐答案

错误消息说明了一切.您在地图中有一个 Any .Spark SQL 和 Dataset api 不支持架构中的 Any.它必须是受支持的类型之一(它是基本类型的列表，例如字符串、整数等.支持类型的序列或支持类型的映射).

The error message says it all. You have an Any in the map. Spark SQL and Dataset api does not support Any in the schema. It has to be one of the supported type (which is a list of basic types such as String, Integer etc. a sequence of supported types or a map of supported types).

这篇关于不支持 Any 类型的架构的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！