当Kryo序列化与Spark一起使用时，ProtoBuf中的NullPointerException [英] NullPointerException in ProtoBuf when Kryo serialization is used with Spark

问题描述

当我尝试序列化protobuf字段时，我的spark应用程序遇到以下错误，该字段是键String和值float的映射.Spark应用程序正在使用Kryo序列化.

I am getting the following error in my spark application when it is trying to serialize a protobuf field which is a map of key String and value float. Kryo serialization is being used in the spark app.

Caused by: java.lang.NullPointerException
    at com.google.protobuf.UnmodifiableLazyStringList.size(UnmodifiableLazyStringList.java:68)
    at java.util.AbstractList.add(AbstractList.java:108)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
    at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:731)
    at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
    ... 71 more

以前有人遇到过这个问题吗?有办法解决吗?

Has anyone faced this issue before? Is there a way to resolve it?

推荐答案

您可以使用kryo注册ProtobufSerializer来序列化protobufs

You can register ProtobufSerializer with kryo to serialize protobufs

• 首先:添加dep:

"de.javakaffee" % "kryo-serializers" % "0.43" // in sbt

• 第二次:扩展kryo序列化器

package com.my.serializer

class ExtendedKryoRegistrator extends KryoRegistrator {
  override def registerClasses(kryo: Kryo): Unit = {
    kryo.register(classOf[YourProtoMessageClass], new ProtobufSerializer())
  }
}

• 第三:使用 ExtendedKryoRegistrator
设置spark conf
• third: set the spark conf with ExtendedKryoRegistrator

val conf = new SparkConf().setAppName("appName")

conf.set("spark.kryo.registrator", "com.my.serializer.ExtendedKryoRegistrator")

        
val spark = SparkSession.builder()
  .config(conf)
  .enableHiveSupport()
  .getOrCreate()

这篇关于当Kryo序列化与Spark一起使用时，ProtoBuf中的NullPointerException的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！