为什么火花任务的单个节点上运行？ [英] Why is the spark task running on a single node?

问题描述

我已经安装了一个Spark集群。它有4个节点，但是当我在集群上运行的SparkPi例子中，只有一个节点运行。我该如何解决呢？

I have installed a Spark cluster. It has 4 nodes, but when I run the SparkPi example on the cluster, there is only one node run. How do I solve it?

推荐答案

一种可能性是，你的运行在本地模式的例子。即，例如不使用簇的。这是很容易验证这一点：停止星火主，看看例子仍然有效。

One possibility is, you are running the example in local mode. That is, the example does not use the cluster at all. It is easy to verify this: stop the Spark master and see if the example still works.

<一个href=\"http://spark.apache.org/docs/latest/submitting-applications.html\">http://spark.apache.org/docs/latest/submitting-applications.html描述如何运行的例子。

http://spark.apache.org/docs/latest/submitting-applications.html describes how to run the example.

bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master local \
  /path/to/examples.jar \
  100

您需要更换本地的东西，如火花：//spark-master.example.com：7077

You need to replace local with something like spark://spark-master.example.com:7077.

另一种解释可能是，只有一个工人在连接以主出于某种原因。或者，在主只分配有限的资源可由于某种原因该应用程序。您可以在主界面检查这两种。 （ http://spark-master.example.com:8080/ ）

Another explanation could be that only one worker has connected to the master for some reason. Or, the master only assigns limited resources to the application for some reason. You can check both of these on the master UI. (http://spark-master.example.com:8080/)

最后，也可以是您的数据被分成太少分区，然后他们都拾起一台机器。在上面的例子中， 100 是分区的数量。如果你忽略它，它默认为 2 。如果你的工人有2个或多个核，它很可能是其中的一种将拿起两个分区。你可以看到，如果是这样的话，当你在应用程序UI的执行阶段。 （的http：//本地主机：4040 /阶段）的解决方案是增加分区的数量

Finally, it can be that your data is split into too few partitions, and they are all picked up by a single machine. In the above example, 100 is the number of partitions. If you omit it, it defaults to 2. If your workers have 2 or more cores, it is likely that one of them will pick up both partitions. You can see if this is the case when you look at the executed stages on the application UI. (http://localhost:4040/stages) The solution is to increase the number of partitions.

这篇关于为什么火花任务的单个节点上运行？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！