Mesos上的Apache Spark:初始作业未接受任何资源 [英] Apache Spark on Mesos: Initial job has not accepted any resources

问题描述

我正在使用Apache Mesos在集群模式下运行Apache Spark.但是，当我启动Spark-Shell运行简单的测试命令(sc.parallelize(0到10，8).count)时，收到以下警告消息:

I am running Apache Spark on cluster mode using Apache Mesos. But, when I start Spark-Shell to run a simple test command (sc.parallelize(0 to 10, 8).count) I receive the following warning message:

16/03/10 11:50:55 WARN TaskSchedulerImpl:初始作业未接受任何资源；正在执行任何操作.检查您的群集用户界面，以确保工作人员已注册并拥有足够的资源

16/03/10 11:50:55 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

如果我检查Mesos WebUI，可以看到Spark-Shell被列为框架，并且列出了一个从属(我自己的机器).对如何解决问题有帮助吗?

If I check on Mesos WebUI I can see that Spark-Shell is listed as a framework and I have listed one slave (my own machine). Any help how to troubleshoot it?

推荐答案

虽然其他大多数答案都集中在spark从属服务器上的资源分配(内核，内存)，但我想强调一下防火墙可能导致完全相同的问题，尤其是在云平台上运行Spark时.

While most of other answers focuses on resource allocation (cores, memory) on spark slaves, I would like to highlight that firewall could cause exactly the same issue, especially when you are running spark on cloud platforms.

如果可以在Web UI中找到spark从属，则可能已经打开了标准端口8080、8081、7077、4040.尽管如此，当您实际运行作业时，它会使用SPARK_WORKER_PORT，spark.driver.port和默认情况下是随机分配的.如果您的防火墙阻止了这些端口，则主服务器无法从从服务器检索任何特定于作业的响应并返回错误.

If you can find spark slaves in the web UI, you have probably opened the standard ports 8080, 8081, 7077, 4040. Nonetheless, when you actually run a job, it uses SPARK_WORKER_PORT, spark.driver.port and spark.blockManager.port which by default are randomly assigned. If your firewall is blocking these ports, the master could not retrieve any job-specific response from slaves and return the error.

您可以通过打开所有端口来运行快速测试，并查看从站是否接受作业.

You can run a quick test by opening all the ports and see whether the slave accepts jobs.

这篇关于Mesos上的Apache Spark:初始作业未接受任何资源的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！