如何在Flink中调试可序列化的异常? [英] How to debug serializable exception in Flink?

问题描述

我遇到了几个可序列化的异常，并且我在Flink的互联网和文档上进行了一些搜索；有一些著名的解决方案，例如transient，extended Serializable等.每次异常的起源都很清楚，但就我而言，我无法找到未序列化的确切位置.

问:我应该如何调试这种异常?

A.scala:

class executor ( val sink: SinkFunction[List[String]] {
    def exe(): Unit = {
        xxx.....addSink(sinks)
    }
}

B.scala:

class Main extends App {
  def createSink: SinkFunction[List[String]] = new StringSink()

  object StringSink {
    // static
    val stringList: List[String] = List()
  }

  // create a testing sink
  class StringSink extends SinkFunction[List[String]] {
    override def invoke(strs: List[String]): Unit = {
        // add strs into the variable "stringList" of the compagin object StringSink
    }
  }

  new executor(createSink()).exe()

  // then do somethings with the strings
}

例外是:

SinkFunction 的实现不可序列化.这 对象可能包含或引用了不可序列化的字段.

我发现了两个可疑点:

1. StringSink的实例被传递到另一个文件中.
2. 在StringSink类中，它使用静态变量stringList 其compagin对象.

解决方案

我遇到了类似的问题.过去需要很长时间才能找出哪些成员/对象不可序列化.异常日志并没有真正的帮助.

帮助我的是以下JVM选项，该选项可以在异常跟踪中启用更多详细信息.

启用此选项...

-Dsun.io.serialization.extendedDebugInfo = true

I've encountered several serializable exceptions, and I did some searching on Flink's internet and doc; there are some famous solutions like transient, extends Serializable etc. Each time the origin of exception is very clear, but in my case, i am unable to find where exactly it is not serialized.

Q: How should i debug this kind of Exception?

A.scala:

class executor ( val sink: SinkFunction[List[String]] {
    def exe(): Unit = {
        xxx.....addSink(sinks)
    }
}

B.scala:

class Main extends App {
  def createSink: SinkFunction[List[String]] = new StringSink()

  object StringSink {
    // static
    val stringList: List[String] = List()
  }

  // create a testing sink
  class StringSink extends SinkFunction[List[String]] {
    override def invoke(strs: List[String]): Unit = {
        // add strs into the variable "stringList" of the compagin object StringSink
    }
  }

  new executor(createSink()).exe()

  // then do somethings with the strings
}

The exception is:

The implementation of the SinkFunction is not serializable. The object probably contains or references non serializable fields.

Two suspicious points that I found:

1. The instance of StringSink is passed into another file.
2. In the class of StringSink, it uses a static variable stringList of its compagin object.

解决方案

I faced similar problems. It used to take longtime to find out what member/object is not serializable. The exception logs are not really helpful.

What helped me is the following JVM option, which enables more details in exception trace.

Enable this option...

-Dsun.io.serialization.extendedDebugInfo=true

这篇关于如何在Flink中调试可序列化的异常?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！