星火任务简单蓄能器不可序列化？ [英] Spark Task not Serializable with simple accumulator?

问题描述

我运行这个简单的code：

I am running this simple code:

val accum = sc.accumulator(0, "Progress");
listFilesPar.foreach {
  filepath =>
    accum += 1
}

listFilesPar是 RDD [字符串]

这引发以下错误：

org.apache.spark.SparkException: Task not serializable

现在我不明白发生了什么
我不把括号，但是括号中，因为我需要写一个漫长的功能。我只是做单元测试

Right now I don't understand what's happening and I don't put parenthesis but brackets because I need to write a lengthy function. I am just doing unit testing

推荐答案

这样做的典型原因是封闭意外捕获的东西。东西，你没有在你的贴有，因为你绝不会想到它会被序列化。

The typical cause of this is that the closure unexpectedly captures something. Something that you did not include in your paste, because you would never expect it would be serialized.

您可以尽量减少你的code，直到你找到它。或者只是打开串行调试日志记录与 -Dsun.io.serialization.extendedDebugInfo = TRUE 。您将在输出中可能看到星火尝试序列化的东西傻了。

You can try to reduce your code until you find it. Or just turn on serialization debug logging with -Dsun.io.serialization.extendedDebugInfo=true. You will probably see in the output that Spark tries to serialize something silly.

这篇关于星火任务简单蓄能器不可序列化？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！