即使在conf中注册了类,Kryo序列化也没有注册 [英] Kryo Serialization not registering even after registering the class in conf

查看:163
本文介绍了即使在conf中注册了类,Kryo序列化也没有注册的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我制作了一个Person类并注册了它,但是在运行时,它显示未注册类.为什么显示这样?

I made a class Person and registered it but on runtime, it shows class not registered.Why is it showing so?

Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 0, not attempting to retry it. Exception during serialization: java.io.IOException: java.lang.IllegalArgumentException: Class is not registered: KyroExample$Person[]
Note: To register this class use: kryo.register(KyroExample$Person[].class);

这是示例代码:

 val conf = new SparkConf().setAppName("kyroExample").setMaster("local")

  conf.set("spark.serializer","org.apache.spark.serializer.KryoSerializer")
  conf.registerKryoClasses(Array(classOf[Person],classOf[String])) //registered the class
  conf.set("spark.kryo.registrationRequired", "true")

  val sparkContext = new SparkContext(conf)

  case class Person(name:String, age:Int) //this is the class

  val personList: immutable.Seq[Person] = (1 to 100000).map(value=> Person(value+"",value))

  val rdd: RDD[Person] = sparkContext.parallelize(personList)

  val evenAge: RDD[Person] = rdd.filter(_.age %2 ==0)

  evenAge.persist(StorageLevel.MEMORY_ONLY_SER)
  evenAge.count()

  evenAge.persist(StorageLevel.MEMORY_ONLY_SER)
  evenAge.count()


  Thread.sleep(200000)

推荐答案

在注册了两个用例之后,它就起作用了:

It worked after registering both use cases :

.registerKryoClasses(
      Array(classOf[Person],classOf[Array[Person]])
    )

这篇关于即使在conf中注册了类,Kryo序列化也没有注册的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆