java.lang.UnsupportedOperationException:不支持 MyClass 类型的架构 [英] java.lang.UnsupportedOperationException: Schema for type MyClass is not supported

问题描述

我使用的是 Spark 1.5.0，但在从我的 rdd 创建数据帧时遇到问题.

I am using Spark 1.5.0 and I have an issue while creating a dataframe from my rdd.

代码如下:

case class MyC (myclass: MyClass) 

  val df = rdd.map {t => MyC(t)}.toDF("cust")
  df.show()

错误信息如下:

线程main"中的异常java.lang.UnsupportedOperationException:不支持 MyClass 类型的架构

Exception in thread "main" java.lang.UnsupportedOperationException: Schema for type MyClass is not supported

对此的任何帮助将不胜感激.

Any help with this will be greatly appreciated.

推荐答案

Spark 使用反射来推断数据帧架构，但不能对任意类这样做.我不确定我是否可以比 Spark 文档更好地陈述答案，并且 有一个部分专门用于此.

Spark uses reflection to infer dataframe schema, but cannot do so for arbitrary classes. I'm not sure if I can state an answer better than the Spark docs can, and there's a section dedicated to exactly this.

具体来说，问题在于 Spark 可以推断模式的类型数量有限.整数、字符串、序列/数组都支持(以及包含这些类型元素的 case 类)，但不支持任意类 MyClass.

To be specific, the problem is that there are a limited number of types for which Spark can infer schema. Ints, Strings, Seqs/Arrays are all supported (as well as case classes containing elements of these types), but an arbitrary class MyClass is not.

这篇关于java.lang.UnsupportedOperationException:不支持 MyClass 类型的架构的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！