AWS EMR错误:作业流中的所有从属终端均已终止 [英] AWS EMR Error : All slaves in the job flow were terminated

问题描述

我正在Amazon AWS上使用Elastic Mapreduce基础架构. jowflow自动终止.根据Amazon Console的最后一个状态更改原因是:作业流中的所有从属均已终止".

I am using Elastic Mapreduce infrastructure on Amazon AWS. A jowflow got terminated automatically. Last state change reason according Amazon Console is : "All slaves in the job flow were terminated".

创建作业流程命令:

elastic-mapreduce --create --name MyCluster --alive --instance-group master --instance-type m1.xlarge --instance-count 1 --bid-price 2.0 --instance-group core --instance-type m1.xlarge --instance-count 10 --bid-price 2.0 --hive-interactive  --enable-debugging

有关工作流程的详细信息:

Details about jobflow :

最近几行日志...

Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Starting Job = job_201310231204_0099, Tracking URL = http://ip-10-197-16-105.us-west-1.compute.internal:9100/jobdetails.jsp?jobid=job_201310231204_0099
Kill Command = /home/hadoop/bin/hadoop job  -Dmapred.job.tracker=10.197.16.105:9001 -kill job_201310231204_0099
2013-10-23 14:11:38,618 Stage-1 map = 0%,  reduce = 0%
2013-10-23 14:11:48,741 Stage-1 map = 100%,  reduce = 0%

正如您在上面的日志中看到的那样，不会引发任何错误.

As you can see above in logs, no error is thrown as such.

我认为的原因是

我认为这是由于现货价格突然上涨的缘故.有关更多详细信息，请参见下面的答案.

I think that this happened because of sudden increase in price od spot instances. More details in my answer below.

推荐答案

我在这里回答我自己的问题.

Here I am answering my own question.

我认为发生这种情况是因为现货实例价格突然上涨.对于m1.xlarge实例，我的出价为每实例每小时2美元.

I think that this happened because of sudden increase in price of spot instances. My bid price was $2 per instance per hour for a m1.xlarge instance.

aws控制台竞价型实例定价的快照:

您会注意到定价出现蓝色跳跃.我的出价是2美元，实际价格跳到11美元.因此，我的集群被自动杀死.

You can notice the blue jumps in pricing. My bid was $2 and the actual price jumped to $11. Hence my cluster was killed automatically.

这篇关于AWS EMR错误:作业流中的所有从属终端均已终止的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！