活动任务是Spark UI中的负数 [英] Active tasks is a negative number in Spark UI

问题描述

使用

你看到的是主动任务是一个负数（总负载与已完成任务的差值）。

这个错误的来源是什么？

---

我拥有很多执行程序的节点。然而，似乎有一项任务似乎已经空闲（我没有看到任何进展），而另一项相同的任务正常完成。

---

这也是相关的： mail 我可以证实很多任务正在创建，因为我使用的是1k或2k的执行程序。

我遇到的错误有点不同：

  16/08/15 20:03:38错误LiveListenerBus：由于事件队列中没有剩余空间，因此丢弃SparkListenerEvent。这可能意味着其中一个SparkListeners速度太慢，无法跟上调度程序启动任务的速度。 
 16/08/15 20:07:18 WARN TaskSetManager：阶段4.0（TID 116652，myfoo.com）中丢失的任务20652.0：FetchFailed（BlockManagerId（61，mybar.com，7337），shuffleId = 0，mapId = 328，reduceId = 20652，message = 
 org.apache.spark.shuffle.FetchFailedException：java.util.concurrent.TimeoutException：超时等待任务。
  
解决方案

这是一个Spark问题，当执行者在失败后重新启动时会出现这个问题，同样的JIRA问题已经被创建。可以从 https://issues.apache.org/jira/browse获取更多相关信息/ SPARK-10141 链接。

When using spark-1.6.2 and pyspark, I saw this:

where you see that the active tasks are a negative number (the difference of the the total tasks from the completed tasks).

What is the source of this error?

---

Node that I have many executors. However, it seems like there is a task that seems to have been idle (I don't see any progress), while another identical task completed normally.

---

Also this is related: that mail I can confirm that many tasks are being created, since I am using 1k or 2k executors.

The error I am getting is a bit different:

16/08/15 20:03:38 ERROR LiveListenerBus: Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the SparkListeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.
16/08/15 20:07:18 WARN TaskSetManager: Lost task 20652.0 in stage 4.0 (TID 116652, myfoo.com): FetchFailed(BlockManagerId(61, mybar.com, 7337), shuffleId=0, mapId=328, reduceId=20652, message=
org.apache.spark.shuffle.FetchFailedException: java.util.concurrent.TimeoutException: Timeout waiting for task.

解决方案

It is a Spark issue. It occurs when executors restart after failures. The JIRA issue for the same is already created. You can get more details about the same from https://issues.apache.org/jira/browse/SPARK-10141 link.

这篇关于活动任务是Spark UI中的负数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！