Spark Kryo:注册自定义序列化程序 [英] Spark Kryo: Register a custom serializer

问题描述

我有一个通过实现com.esotericsoftware.kryo.Serializer中的read()和write()方法来实现自定义Kryo序列化程序的类(请参见下面的示例). 如何向Spark注册此自定义序列化程序?

I have a class that implements a custom Kryo serializer by implementing the read() and write() methods from com.esotericsoftware.kryo.Serializer (see example below). How can I register this custom serializer with Spark?

这是我所拥有的伪代码示例:

Here is a pseudo-code example of what I have:

class A() 

CustomASerializer extends com.esotericsoftware.kryo.Serializer[A]{
    override def write(kryo: Kryo, output: Output, a: A): Unit = ???
    override def read(kryo: Kryo, input: Input, t: Class[A]): A = ???
}

val kryo: Kryo = ... 
kryo.register(classOf[A], new CustomASerializer()); // I can register my serializer

现在在Spark中:

val sparkConf = new SparkConf()
sparkConf.registerKryoClasses(Array(classOf[A]))

不幸的是，Spark没有给我注册我的自定义序列化程序的选项.知道是否有办法做到这一点吗?

Unfortunately, Spark doesn't give me the option to register my custom serializer. Any idea if there is a way to do this?

推荐答案

使用以下已注册的自定义序列化程序创建自己的 KryoRegistrator :

Create your own KryoRegistrator with this custom serializer registered:

package com.acme

class MyRegistrator extends KryoRegistrator {
  override def registerClasses(kryo: Kryo) {
    kryo.register(classOf[A], new CustomASerializer())
  } 
}

然后，将spark.kryo.registrator设置为注册人的全名，例如com.acme.MyRegistrator:

Then, set spark.kryo.registrator to your registrator's fully-qualified name, e.g. com.acme.MyRegistrator:

val conf = new SparkConf()
conf.set("spark.kryo.registrator", "com.acme.KryoRegistrator")

这篇关于Spark Kryo:注册自定义序列化程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！