Flink Kryo 序列化器，因为找不到 Chill 序列化器 [英] Flink Kryo serializer because Chill serializer couldn't be found

问题描述

有时会引发此错误并关闭任务管理器.我使用了 org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode

Sometimes raise this error and go to down the task manager. I used org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode

WARN  org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer  - Falling back to default Kryo serializer because Chill serializer couldn't be found.
    java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.getKryoInstance(KryoSerializer.java:436)
        at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.checkKryoInitialized(KryoSerializer.java:454)
        at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:337)
        at org.apache.flink.api.common.typeutils.CompositeSerializer.deserialize(CompositeSerializer.java:151)
        at org.apache.flink.contrib.streaming.state.ttl.RocksDbTtlCompactFiltersManager$ListElementFilter.nextElementLastAccessTimestamp(RocksDbTtlCompactFiltersManager.java:202)
        at org.apache.flink.contrib.streaming.state.ttl.RocksDbTtlCompactFiltersManager$ListElementFilter.nextUnexpiredOffset(RocksDbTtlCompactFiltersManager.java:189)
    Caused by: java.lang.ClassNotFoundException: scala/Function0
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at com.twitter.chill.KryoBase.$anonfun$functions$1(KryoBase.scala:41)
        at com.twitter.chill.KryoBase.$anonfun$functions$1$adapted(KryoBase.scala:41)
        at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
        at scala.collection.immutable.Range.foreach(Range.scala:155)
        at scala.collection.TraversableLike.map(TraversableLike.scala:233)
        at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
        at scala.collection.AbstractTraversable.map(Traversable.scala:104)
        at com.twitter.chill.KryoBase.<init>(KryoBase.scala:41)
        at org.apache.flink.runtime.types.EmptyFlinkScalaKryoInstantiator.newKryo(FlinkScalaKryoInstantiator.scala:45)
        at org.apache.flink.runtime.types.FlinkScalaKryoInstantiator.newKryo(FlinkScalaKryoInstantiator.scala:82)
        ... 10 more
    Exception in thread "Thread-9" java.lang.IllegalArgumentException: classLoader cannot be null.
        at com.esotericsoftware.kryo.Kryo.setClassLoader(Kryo.java:975)
        at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.checkKryoInitialized(KryoSerializer.java:477)
        at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:337)
        at org.apache.flink.api.common.typeutils.CompositeSerializer.deserialize(CompositeSerializer.java:151)
        at org.apache.flink.contrib.streaming.state.ttl.RocksDbTtlCompactFiltersManager$ListElementFilter.nextElementLastAccessTimestamp(RocksDbTtlCompactFiltersManager.java:202)
        at org.apache.flink.contrib.streaming.state.ttl.RocksDbTtlCompactFiltersManager$ListElementFilter.nextUnexpiredOffset(RocksDbTtlCompactFiltersManager.java:189)

推荐答案

这可能是一个 问题在本机压缩线程的上下文中没有用户类加载器.

This is probably an issue of not having user class loader in the context of the native compaction thread.

作为一种解决方法，我建议使用带有简单元素的列表状态，这些元素类型在序列化时具有固定的字节长度.那将是原始类型和 POJO，如果可能的话，可能会实现自定义的固定长度序列化程序.那么 Kryo 序列化器就不必在 TTL 清理中调用，并且需要用户类加载器.

As a workaround, I would suggest to use a list state with simple elements which type has a fixed byte length when serialized. That would be primitive types and POJOs, maybe implement a custom fixed length serializer if possible. Then the Kryo serializer does not have to be called in the TTL cleanup and requires the user class loader.

这篇关于Flink Kryo 序列化器，因为找不到 Chill 序列化器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！