TaskSchedulerImpl：初始作业尚未接受任何资源; [英] TaskSchedulerImpl: Initial job has not accepted any resources;

问题描述

这是我想要做的。

我已经创建了两个DataStax企业集群节点，其中我创建了一个java程序，计数一个表（Cassandra数据库表）。

I have created two nodes of DataStax enterprise cluster,on top of which I have created a java program to get the count of one table (Cassandra database table).

这个程序是在eclipse中构建的，它实际上是从一个窗口框。

This program was built in eclipse which is actually from a windows box.

在从Windows运行此程序时，它在运行时失败，出现以下错误：

At the time of running this program from windows it's failing with the following error at runtime:

初始工作未接受任何资源;检查您的集群UI以确保工作程序已注册并具有足够的内存

Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

在这些集群上成功运行，没有任何问题。

The same code has been compiled & run successfully on those clusters without any issue. What could be the reason why am getting above error?

代码：

import org.apache.spark.SparkConf;

import org.apache.spark.SparkContext;

import org.apache.spark.api.java.JavaSparkContext;
import org.apache.spark.sql.SchemaRDD;
import org.apache.spark.sql.cassandra.CassandraSQLContext;
import com.datastax.bdp.spark.DseSparkConfHelper;

public class SparkProject  {

    public static void main(String[] args) {

        SparkConf conf = DseSparkConfHelper.enrichSparkConf(new SparkConf()).setMaster("spark://10.63.24.14X:7077").setAppName("DatastaxTests").set("spark.cassandra.connection.host","10.63.24.14x").set("spark.executor.memory", "2048m").set("spark.driver.memory", "1024m").set("spark.local.ip","10.63.24.14X");

        JavaSparkContext sc = new JavaSparkContext(conf);

        CassandraSQLContext cassandraContext = new CassandraSQLContext(sc.sc());
        SchemaRDD employees = cassandraContext.sql("SELECT * FROM portware_ants.orders");

        //employees.registerTempTable("employees");
        //SchemaRDD managers = cassandraContext.sql("SELECT symbol FROM employees");
        System.out.println(employees.count());

        sc.stop();
    }
}

推荐答案

I面对类似的问题，经过一些在线研究和试错误，我缩小到3个原因（除了第一个，其他两个甚至不接近错误消息）：

I faced similar issue and after some online research and trial-n-error, I narrowed down to 3 causes for this (except for the first the other two are not even close to the error message):

1. 如错误所示，可能你分配的资源超过了可用。 =>这不是我的问题


2. 主机名和IP地址错误：通过在spark-env.sh中指定SPARK_MASTER_IP和SPARK_LOCAL_IP


3. 在客户端上禁用防火墙是为我工作的解决方案。因为我在一个原型内部代码，我禁用客户端节点上的防火墙。由于某些原因，工作节点，无法为我回答客户端。出于生产目的，您将需要打开一定数量的端口。

1. As indicated by the error, probably you are allocating the resources more than that is available. => This was not my issue
2. Hostname & IP Address mishaps: I took care of this by specifying the SPARK_MASTER_IP and SPARK_LOCAL_IP in spark-env.sh
3. Disable Firewall on the client : This was the solution that worked for me. Since I was working on a prototype in-house code, I disabled the firewall on the client node. For some reason the worker nodes, were not able to talk back to the client for me. For production purposes, you would want to open-up certain number of ports required.

这篇关于TaskSchedulerImpl：初始作业尚未接受任何资源;的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！