Spark 作业完成但应用程序需要时间关闭 [英] Spark jobs finishes but application takes time to close

问题描述

使用 scala 运行 spark 作业，正如预期的那样，所有作业都按时完成，但不知何故，一些 INFO 日志在作业停止前打印了 20-25 分钟.

Running spark job using scala, as expected all jobs are finishing up on time , but somehow some INFO logs are printed for 20-25 minutes before job stops.

贴几张UI截图，可以帮助理解问题.

Posting few UI screenshot which can help to undestand the problem .

1. 以下是 4 个阶段所花费的时间:

1. 以下是连续作业 ID 之间的时间

我不明白为什么在两个作业 ID 之间花费了这么多时间.

I dont understand why there is so much time spent in between both job ids.

以下是我的代码片段:

    val sc = new SparkContext(conf)
for (x <- 0 to 10) {
  val zz = getFilesList(lin);
  val links = zz._1
  val path = zz._2
  lin = zz._3
  val z = sc.textFile(links.mkString(",")).map(t => t.split('	')).filter(t => t(4) == "xx" && t(6) == "x").map(t => titan2(t)).filter(t => t.length > 35).map(t => ((t(34)), (t(35), t(5), t(32), t(33))))
  val way_nodes = sc.textFile(way_source).map(t => t.split(";")).map(t => (t(0), t(1)));
  val t = z.join(way_nodes).map(t => (t._2._1._2, Array(Array(t._2._1._2, t._2._1._3, t._2._1._4, t._2._1._1, t._2._2)))).reduceByKey((t, y) => t ++ y).map(t => process(t)).flatMap(t => t).combineByKey(createTimeCombiner, timeCombiner, timeMerger).map(averagingFunction).map(t => t._1 + "," + t._2)
  t.saveAsTextFile(path)
}
sc.stop()

更多后续:spark-1.4.1 saveAsTextFile 到 S3 在 emr-4.0.0 上很慢

推荐答案

我最终升级了我的 Spark 版本并且问题得到了解决.

I ended up upgrading my spark version and issue was resolved .

这篇关于Spark 作业完成但应用程序需要时间关闭的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！