不支持任何类型的架构 [英] Schema for type Any is not supported

问题描述

我正在尝试创建一个火花UDF，以从用户定义的案例类中提取(键，值)对的映射.

I'm trying to create a spark UDF to extract a Map of (key, value) pairs from a User defined case class.

scala函数似乎工作正常，但是当我尝试在spark2.0中将其转换为UDF时，遇到了不支持Any类型的模式"错误.

The scala function seems to work fine, but when I try to convert that to a UDF in spark2.0, I'm running into the " Schema for type Any is not supported" error.

case class myType(c1: String, c2: Int)
def getCaseClassParams(cc: Product): Map[String, Any] = {

    cc
      .getClass
      .getDeclaredFields // all field names
      .map(_.getName)
      .zip(cc.productIterator.to) // zipped with all values
      .toMap

  }

但是当我尝试将函数值实例化为UDF时，会导致以下错误-

But when I try to instantiate a function value as a UDF it results in the following error -

val ccUDF = udf{(cc: Product, i: String) => getCaseClassParams(cc).get(i)}

java.lang.UnsupportedOperationException: Schema for type Any is not supported
  at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:716)
  at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:668)
  at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:654)
  at org.apache.spark.sql.functions$.udf(functions.scala:2841)

推荐答案

错误消息说明了一切.您在地图上有一个Any. Spark SQL和Dataset API在架构中不支持Any.它必须是一种受支持的类型(它是诸如String，Integer等基本类型的列表.受支持的类型的序列或受支持的类型的映射).

The error message says it all. You have an Any in the map. Spark SQL and Dataset api does not support Any in the schema. It has to be one of the supported type (which is a list of basic types such as String, Integer etc. a sequence of supported types or a map of supported types).

这篇关于不支持任何类型的架构的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！