使用Spark，如何连接主或解决错误：“WARN TaskSchedulerImpl：初始作业尚未接受任何资源” [英] With Spark,how to connect master or solve an error:"WARN TaskSchedulerImpl: Initial job has not accepted any resources"

问题描述

请告诉我如何跟随问题。

首先，我确认下面的代码在master是local时运行。

然后我启动了两个EC2实例（m1.large）。
但是，当master为spark：// MASTER_PUBLIC_DNS：7077时，将显示错误消息TaskSchedulerImpl，并且失败。

当我从VALID地址更改为无效地址（spark：// INVALID_DNS：7077）时，会出现相同的错误消息。

即WARN TaskSchedulerImpl：初始作业尚未接受任何资源;请检查您的群集用户界面以确保工作人员已注册且具有足够内存

<看起来像这个。
作为此评论，我为此群集分配了12G内存，但它失败了。
$ b

 ＃！/ usr / bin / env python 
＃ -  *  -  coding：utf-8  -  *  -  
来自pyspark import SparkContext，SparkConf $ b $ p from pyspark.mllib.classification import LogisticRegressionWithSGD $ b $ from pyspark.mllib.regression从numpy导入数组中导入LabeledPoint 
 
 
＃加载并解析数据
 def parsePoint（line）：
 values = [float（x）for x in line。 split（''）] 
 return LabeledPoint（values [0]，values [1：]）
 appName =testsparkapp
 master =spark：// MASTER_PUBLIC_DNS：7077
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 data = sc.textFile（/ root / spark / mllib / data / sample_svm_data.txt）
 parsedData = data.map（parsePoint）
 
＃构建模型
 model = Logist icRegressionWithSGD.train（parsedData）
 
＃在训练数据上评估模型
 labelsAndPreds = parsedData.map（lambda p：（p.label，model.predict（p.features）））
 trainErr = labelsAndPreds.filter（lambda（v，p）：v！= p）.count（）/ float（parsedData.count（））
 print（Training Error =+ str（trainErr） ）
  

其他

我的朋友告诉我。

1.我打开了主端口7077.

2.在master url，设置主机名不是ip地址。

- >因此，我开始能够连接主服务器（我通过Cluster UI进行了检查）。

3.我尝试设置worker_max_heap，如下所示，但可能会失败。

ScalaConf（）.set（spark.executor.memory，4g ）.set（worker_max_heapsize，2g）

工作人员允许使用6.3GB（我通过UI检查过）。它是m1.large。

- >我在执行日志中发现了一个警告，并在工作人员stderr中发生了错误。我的执行日志中有

  14/08/08 06:11:59 WARN TaskSchedulerImpl：初始作业未接受任何资源;检查你的集群用户界面，以确保工人已注册并有足够的内存
  

worker stderr

  14/08/08 06:14:04信息worker.WorkerWatcher：已成功连接到akka.tcp：// sparkWorker @ PRIVATE_HOST_NAME1：52011 / user / Worker 
 14/08/08 06:15:07 ERROR executor.CoarseGrainedExecutorBackend：Driver Disassociated [akka.tcp：// sparkExecutor @ PRIVATE_HOST_NAME1：52201]  - > [akka.tcp：// spark @ PRIVATE_HOST_NAME2：38286]解除关联！关机。 
  

解决方案

spark-ec2脚本配置EC2中的Spark Cluster作为独立的，这意味着它不能使用远程提交。我一直在与你之前描述的同样的错误挣扎，然后才发现它不被支持。不幸的是，消息错误是不正确的。

所以你必须复制你的东西并登录到master来执行你的spark任务。

Please tell me to how to following problem.

Firstly,I confirmed that following code run when master is "local".

Then I started two EC2 instances(m1.large). However,when master is "spark://MASTER_PUBLIC_DNS:7077",the error message,"TaskSchedulerImpl", appears and it fails.

When I change to INVALID address as a master(spark://INVALID_DNS:7077) from VALID address,the same error message appears.

Namely,"WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory"

It seems like this. As this comment, I assigned 12G memory to this cluster but it fails.

#!/usr/bin/env python                                                                                     
# -*- coding: utf-8 -*- 
from pyspark import SparkContext, SparkConf 
from pyspark.mllib.classification import LogisticRegressionWithSGD 
from pyspark.mllib.regression import LabeledPoint 
from numpy import array 

# Load and parse the data 
def parsePoint(line): 
  values = [float(x) for x in line.split(' ')] 
  return LabeledPoint(values[0], values[1:]) 
appName = "testsparkapp" 
master = "spark://MASTER_PUBLIC_DNS:7077" 
#master = "local" 


conf = SparkConf().setAppName(appName).setMaster(master) 
sc = SparkContext(conf=conf) 

data = sc.textFile("/root/spark/mllib/data/sample_svm_data.txt") 
parsedData = data.map(parsePoint) 

# Build the model 
model = LogisticRegressionWithSGD.train(parsedData) 

# Evaluating the model on training data 
labelsAndPreds = parsedData.map(lambda p: (p.label, model.predict(p.features))) 
trainErr = labelsAndPreds.filter(lambda (v, p): v != p).count() / float(parsedData.count()) 
print("Training Error = " + str(trainErr))     

Additional

I did three tasks that my friend adviced me to.

1.I opened master port,7077.

2.In master url, set host name not ip address.

->There fore,I became to able to connect master server(I checked it by Cluster UI).

3.I tried to set worker_max_heap,as following, but it fails probably.

ScalaConf().set("spark.executor.memory", "4g").set("worker_max_heapsize","2g")

the worker allow me to use 6.3GB(I checked it by UI).It is m1.large.

->I recognized a warning in my executing log and an error in a worker stderr.

my executing log

14/08/08 06:11:59 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

worker stderr

14/08/08 06:14:04 INFO worker.WorkerWatcher: Successfully connected to akka.tcp://sparkWorker@PRIVATE_HOST_NAME1:52011/user/Worker
14/08/08 06:15:07 ERROR executor.CoarseGrainedExecutorBackend: Driver Disassociated [akka.tcp://sparkExecutor@PRIVATE_HOST_NAME1:52201] -> [akka.tcp://spark@PRIVATE_HOST_NAME2:38286] disassociated! Shutting down.

解决方案

The spark-ec2 script configure the Spark Cluster in EC2 as standalone, which mean it can not work with remote submits. I've been struggled with this same error you described for days before figure out it's not supported. The message error is unfortunately incorrect.

So you have to copy your stuff and log into the master to execute your spark task.

这篇关于使用Spark，如何连接主或解决错误：“WARN TaskSchedulerImpl：初始作业尚未接受任何资源”的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！