无法执行不止一个火花作业“初始作业未接受任何资源" [英] Unable to Execute More than a spark Job "Initial job has not accepted any resources"

问题描述

使用独立的Spark Java执行以下代码段，我得到的状态始终在等待，并显示以下错误.当我尝试添加Print语句时，它不起作用.我可能错过了运行多个作业的任何配置吗?

Using a Standalone Spark Java to execute the below code snippet, I'm getting the Status is always WAITING with the below error.It doesn't work when I try to add the Print statement. Is there any configuration I might have missed to run multiple jobs?

15/09/18 15:02:56信息DAScheduler:从阶段0提交2个丢失的任务(MapPartitionsRDD [2]在SparkTest.java:143处的过滤器中)

15/09/18 15:02:56 INFO DAGScheduler: Submitting 2 missing tasks from Stage 0 (MapPartitionsRDD[2] at filter at SparkTest.java:143)

15/09/18 15:02:56 INFO TaskSchedulerImpl:使用2添加任务集0.0任务

15/09/18 15:02:56 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks

15/09/18 15:03:11警告TaskSchedulerImpl:初始作业未接受任何资源；检查您的集群用户界面，以确保工作人员注册并有足够的资源

15/09/18 15:03:11 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/09/18 15:03:26 WARN TaskSchedulerImpl:初始作业未接受任何资源；检查您的集群用户界面，以确保工作人员注册并有足够的资源

15/09/18 15:03:26 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/09/18 15:03:41警告TaskSchedulerImpl:初始作业未接受任何资源；检查您的集群用户界面，以确保工作人员注册并有足够的资源

15/09/18 15:03:41 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

JavaRDD<String> words = input.flatMap(new FlatMapFunction<String, String>()    //Ln:143
        {
            public Iterable<String> call(String x)
            {
                return Arrays.asList(x.split(" "));
            }
        });
// Count all the words
System.out.println("Total words is" + words.count())

推荐答案

此错误消息表示，您的应用程序从群集请求的资源比群集当前提供的资源更多，即，内核或内存比群集中提供的更多.

This error message means that your application is requesting more resources from the cluster than the cluster can currently provide i.e. more cores or more RAM than available in the cluster.

造成这种情况的原因之一可能是您已经在运行一项作业，该作业占用了所有可用的内核.

One of the reasons for this could be that you already have a job running which uses up all the available cores.

发生这种情况时，您的工作很可能正在等待另一项工作完成并释放资源.

When this happens, your job is most probably waiting for another job to finish and release resources.

您可以在Spark UI中进行检查.

You can check this in the Spark UI.

这篇关于无法执行不止一个火花作业“初始作业未接受任何资源"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！