避免“任务不可序列化".在类中使用嵌套方法 [英] Avoid "Task not serialisable" with nested method in a class

问题描述

我了解访问字段或闭包范围之外的方法时通常会出现的无法序列化任务"问题.

I understand the usual "Task not serializable" issue that arises when accessing a field or a method that is out of scope of a closure.

要解决此问题，我通常定义这些字段/方法的本地副本，从而避免了序列化整个类的需要:

To fix it, I usually define a local copy of these fields/methods, which avoids the need to serialize the whole class:

class MyClass(val myField: Any) { 
  def run() = { 
    val f = sc.textFile("hdfs://xxx.xxx.xxx.xxx/file.csv") 

    val myField = this.myField 
    println(f.map( _ + myField ).count) 
  } 
} 

现在，如果我在run方法中定义了一个嵌套函数，则无法将其序列化:

Now, if I define a nested function in the run method, it cannot be serialized:

class MyClass() { 
  def run() = { 
    val f = sc.textFile("hdfs://xxx.xxx.xxx.xxx/file.csv") 

    def mapFn(line: String) = line.split(";") 

    val myField = this.myField 
    println(f.map( mapFn( _ ) ).count) 

  } 
} 

我不明白，因为我认为"mapFn"应该在范围之内...甚至更陌生，如果我将mapFn定义为val而不是def，那么它将起作用:

I don't understand since I thought "mapFn" would be in scope... Even stranger, if I define mapFn to be a val instead of a def, then it works:

class MyClass() { 
  def run() = { 
    val f = sc.textFile("hdfs://xxx.xxx.xxx.xxx/file.csv") 

    val mapFn = (line: String) => line.split(";") 

    println(f.map( mapFn( _ ) ).count)     
  } 
} 

这与Scala表示嵌套函数的方式有关吗?

Is this related to the way Scala represents nested functions?

处理此问题的推荐方法是什么?避免嵌套函数?

What's the recommended way to deal with this issue ? Avoid nested functions?

推荐答案

它不是以这种方式工作的，因此在第一种情况下， f.map(mapFN(_))等效于 f.map(new Function(){override def apply(...)= mapFN(...)})，在第二个代码中，它只是 f.map(mapFN)?当使用 def 声明方法时，它可能仅仅是某些匿名类中的方法，该类具有对封闭类的隐式 $ outer 引用.但是 map 需要一个 Function ，因此编译器需要包装它.在包装器中，您仅引用该匿名类的某些方法，而不引用实例本身.如果使用 val ，则直接引用传递给 map 的函数.我不确定，只是大声思考...

Isn't it working in the way so that in the first case f.map(mapFN(_)) is equivalent to f.map(new Function() { override def apply(...) = mapFN(...) }) and in the second one it is just f.map(mapFN)? When you declare a method with def it is probably just a method in some anonymous class with implicit $outer reference to the enclosing class. But map requires a Function so the compiler needs to wrap it. In the wrapper you just refer to some method of that anonymous class, but not to the instance itself. If you use val, you have a direct reference to the function which you pass to the map. I'm not sure about this, just thinking out loud...

这篇关于避免“任务不可序列化".在类中使用嵌套方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！