Spark Kryo注册数组类 [英] Spark Kryo register for array class

问题描述

我正在尝试使用数组注册一个类(激活了Kryo的Spark Java)，日志显示了一条清晰的消息:

I am trying to register a class with array (Spark Java with Kryo activated), log shows a clear message:

Class is not registered: org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[]

我写了几种组合，但是这些组合不起作用:

I have written several combinations, but these do not work:

        kryo.register(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[]")); // ERROR
        kryo.register(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[].class")); // ERROR
        kryo.register(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation$Array")); // ERROR
        kryo.register(Class.forName("[org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation")); // ERROR
        kryo.register(Class.forName("[Lorg.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation")); // ERROR
        kryo.register(Class.forName("Array[org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation]")); // ERROR
        kryo.register(Class.forName("[[org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation"));  // ERROR

我还尝试编写没有 Class.forName 的注册类，但是Java无法解析符号 InMemoryFileIndex $ SerializableBlockLocation :

I also tried to write a registration class without Class.forName but Java cannot resolve the symbol InMemoryFileIndex$SerializableBlockLocation:

kryo.register(org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[].class);

所有其他类都在我的KryoRegister.class中工作.

All other classes work in my KryoRegister.class.

推荐答案

我发现了一个类似的问题

I found a similar question here, try:

kryo.register(Array.newInstance(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation"), 0)
                        .getClass());

这篇关于Spark Kryo注册数组类的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！