java.lang.UnsupportedOperationException:不支持类型MyClass的架构 [英] java.lang.UnsupportedOperationException: Schema for type MyClass is not supported

问题描述

我正在使用Spark 1.5.0，从rdd创建数据帧时遇到问题.

I am using Spark 1.5.0 and I have an issue while creating a dataframe from my rdd.

这是代码:

case class MyC (myclass: MyClass) 

  val df = rdd.map {t => MyC(t)}.toDF("cust")
  df.show()

这是错误消息:

线程"main"中的异常java.lang.UnsupportedOperationException: 不支持MyClass类型的架构

Exception in thread "main" java.lang.UnsupportedOperationException: Schema for type MyClass is not supported

任何帮助，将不胜感激.

Any help with this will be greatly appreciated.

推荐答案

Spark使用反射来推断数据帧架构，但不能对任意类这样做.我不确定我能否说出比Spark文档更好的答案，并且

Spark uses reflection to infer dataframe schema, but cannot do so for arbitrary classes. I'm not sure if I can state an answer better than the Spark docs can, and there's a section dedicated to exactly this.

具体来说，问题在于Spark可以推断模式的类型数量有限.整数，字符串，序列/数组均受支持(以及包含这些类型的元素的case类)，但不支持任意类MyClass.

To be specific, the problem is that there are a limited number of types for which Spark can infer schema. Ints, Strings, Seqs/Arrays are all supported (as well as case classes containing elements of these types), but an arbitrary class MyClass is not.

这篇关于java.lang.UnsupportedOperationException:不支持类型MyClass的架构的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！