为什么从不触发SparkListenerApplicationStart? [英] Why is SparkListenerApplicationStart never fired?

问题描述

我正在编写一个Spark应用程序，我需要拦截正在运行的作业的状态.为此，我使用以下代码实现了SparkListener:

I am writing a Spark application and I need to intercept the status of the running jobs. I implemented a SparkListener for this purpose, using the following code:

class MyAppListener extends SparkListener {

    override def onApplicationStart(ev: SparkListenerApplicationStart): Unit = {
      println("AAA: Application Start")
    }

    override def onApplicationEnd(ev: SparkListenerApplicationEnd): Unit = {
      println("AAA: Application End")
    }
  }
}

然后，我使用以下代码启动应用程序并查看事件:

Then, I used the following code to start the application and see the events:

val appListener = new MyAppListener
val conf = new SparkConf().setAppName("Listener")
val sc = new SparkContext(conf) 
sc.addSparkListener(appListener)
println(sc.parallelize(1 to 10).count)
sc.stop()

在日志中，我看到字符串"AAA:应用程序结束"，但没有看到应用程序的开始.

In the logs, I see the string "AAA: Application End", but I don't see the start of the application.

配置:

• Spark版本1.2.0
• Scala 2.10
• 集群上的本地独立模式或YARN客户端模式(相同的行为)

推荐答案

您正在将侦听器添加到错误的位置，当您启动spark上下文时，它也会启动您的应用程序.=>至此，您已经添加了侦听器，onApplicationStart已被触发.

You were adding your listener to spark in the wrong place, When you initiate a spark context, it also starts your application.=> At the point you added your listener, onApplicationStart has already been fired.

解决方案:将您的侦听器添加到SparkConf.

Solution: Add your listener to SparkConf.

sparkConf.set("spark.extraListeners","your.listener.class")

这篇关于为什么从不触发SparkListenerApplicationStart?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！