org.apache.spark.SparkException:任务不可序列化 [英] org.apache.spark.SparkException: Task not serializable

问题描述

这是一个工作代码示例:

This is a working code example:

JavaPairDStream<String, String> messages = KafkaUtils.createStream(javaStreamingContext, zkQuorum, group, topicMap);
messages.print();
JavaDStream<String> lines = messages.map(new Function<Tuple2<String, String>, String>() {
    @Override
    public String call(Tuple2<String, String> tuple2) {
        return tuple2._2();
    }
});

我收到以下错误:

ERROR:
org.apache.spark.SparkException: Task not serializable
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:166)
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:158)
    at org.apache.spark.SparkContext.clean(SparkContext.scala:1435)
    at org.apache.spark.streaming.dstream.DStream.map(DStream.scala:438)
    at org.apache.spark.streaming.api.java.JavaDStreamLike$class.map(JavaDStreamLike.scala:140)
    at org.apache.spark.streaming.api.java.JavaPairDStream.map(JavaPairDStream.scala:46)

推荐答案

由于您使用匿名内部类定义地图函数，因此包含类也必须是可序列化的.将您的地图函数定义为单独的类或使其成为静态内部类.从 Java 文档 (http://docs.oracle.com/javase/8/docs/platform/serialization/spec/serial-arch.html):

Since you're defining your map function using an anonymous inner class, the containing class must also be Serializable. Define your map function as a separate class or make it a static inner class. From the Java documentation (http://docs.oracle.com/javase/8/docs/platform/serialization/spec/serial-arch.html):

注意 - 内部类(即不是静态成员类的嵌套类)的序列化，包括本地和匿名类，由于多种原因，强烈建议不要使用.因为在非静态上下文中声明的内部类包含对封闭类实例的隐式非瞬态引用，所以序列化这样的内部类实例也会导致其关联的外部类实例的序列化.

Note - Serialization of inner classes (i.e., nested classes that are not static member classes), including local and anonymous classes, is strongly discouraged for several reasons. Because inner classes declared in non-static contexts contain implicit non-transient references to enclosing class instances, serializing such an inner class instance will result in serialization of its associated outer class instance as well.

这篇关于org.apache.spark.SparkException:任务不可序列化的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！