为什么出现错误“无法找到存储在数据集中的类型的编码器"?使用案例类编码JSON时? [英] Why is the error "Unable to find encoder for type stored in a Dataset" when encoding JSON using case classes?

问题描述

我写了火花工作:

object SimpleApp {
  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("Simple Application").setMaster("local")
    val sc = new SparkContext(conf)
    val ctx = new org.apache.spark.sql.SQLContext(sc)
    import ctx.implicits._

    case class Person(age: Long, city: String, id: String, lname: String, name: String, sex: String)
    case class Person2(name: String, age: Long, city: String)

    val persons = ctx.read.json("/tmp/persons.json").as[Person]
    persons.printSchema()
  }
}

在IDE中运行主函数时，发生2个错误:

In IDE when I run the main function, 2 error occurs:

Error:(15, 67) Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing sqlContext.implicits._  Support for serializing other types will be added in future releases.
    val persons = ctx.read.json("/tmp/persons.json").as[Person]
                                                                  ^

Error:(15, 67) not enough arguments for method as: (implicit evidence$1: org.apache.spark.sql.Encoder[Person])org.apache.spark.sql.Dataset[Person].
Unspecified value parameter evidence$1.
    val persons = ctx.read.json("/tmp/persons.json").as[Person]
                                                                  ^

但是在Spark Shell中，我可以运行此作业而没有任何错误.有什么问题吗?

but in Spark Shell I can run this job without any error. what is the problem?

推荐答案

错误消息指出Encoder无法采用Person case类.

The error message says that the Encoder is not able to take the Person case class.

Error:(15, 67) Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing sqlContext.implicits._  Support for serializing other types will be added in future releases.

将案例类的声明移至SimpleApp范围之外.

Move the declaration of the case class outside the scope of SimpleApp.

这篇关于为什么出现错误“无法找到存储在数据集中的类型的编码器"?使用案例类编码JSON时?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！