hadoop-streaming：在待处理状态下减少任务，表示“没有空间来减少任务”。 [英] hadoop-streaming : reduce task in pending state says "No room for reduce task."

问题描述

我的地图任务成功完成，我可以看到应用程序日志，但是reducer保持未决状态

 种类％Complete Num任务等待运行完成死亡失败/杀死任务尝试
地图100.00％200 0 0 200 0 0/40 
减少0.00％1 1 0 0 0 0/0 
  

当我看到reduce任务时，我看到

 所有任务尝试
 
无任何尝试发现
  

当我看到hadoop-hduser-jobtracker-master.log，我看到以下内容：

  2011-10-31 00：00：00,238 WARN org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_worker2：localhost / 127.0.0.1：35984有67730714624字节空闲;但我们预计需要减少输入184825113602 
 WARNING org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_master：localhost / 127.0.0.1：43078有68041449472字节空闲;但我们预计需要减少输入184825113602 
 WARNING org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_worker1：localhost / 127.0.0.1：34794有67397398528字节空闲;但我们预计需要减少输入184825113602 
 2011-10-31 00：00：04,532 WARN org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_worker2：localhost / 127.0.0.1：35984有67729956864字节空闲;但我们预计需要减少输入184825113602 
 2011-10-31 00：00：04,695 WARN org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_master：localhost / 127.0.0.1：43078有67998732288字节空闲;但我们预计需要减少输入184825113602 
 2011-10-31 00：00：06,865 WARN org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_worker1：localhost / 127.0.0.1：34794有67397390336字节空闲;但我们预计减少输入需要184825113602 
 WARNING org.apache.hadoop.mapred.JobInProgress：没有空间用于减少任务。节点tracker_master：localhost / 127.0.0.1：43078有67948941312字节空闲;但我们预计减少投入需要184825113602 
 WAR WARN org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_worker2：localhost / 127.0.0.1：35984有67728035840字节空闲;但我们预计需要减少投入184825113602 
 2011-10-31 00：00：11,118 WARN org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_master：localhost / 127.0.0.1：43078有67907883008字节空闲;但是我们预计需要减少输入184825113602 
 2011-10-31 00：00：12,258 WARN org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_worker1：localhost / 127.0.0.1：34794有67397365760字节空闲;但我们预计需要减少输入184825113602 
 2011-10-31 00：00：14,119 WARN org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_master：localhost / 127.0.0.1：43078有67867385856字节空闲;但我们预计需要减少输入184825113602 
 2011-10-31 00：00：16,061 WARN org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_worker1：localhost / 127.0.0.1：34794有67397390336字节空闲;但我们预计需要减少输入184825113602 
 2011-10-31 00：00：17,120 WARN org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_master：localhost / 127.0.0.1：43078有67813683200字节空闲;但我们预计需要减少输入184825113602 
 WARNING org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_worker2：localhost / 127.0.0.1：35984有67726835712字节空闲;但我们预计需要减少输入184825113602 
 2011-10-31 00:00:00 20,122 WARN org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_master：localhost / 127.0.0.1：43078有67896156160字节空闲;但我们预计需要减少投入184825113602 
 WARNING org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_worker1：localhost / 127.0.0.1：34794有67397390336字节空闲;但我们预计需要减少投入184825113602 
 WARNING org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_master：localhost / 127.0.0.1：43078有68150616064字节空闲;但我们预计需要减少输入184825113602 
 WARNING org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_worker1：localhost / 127.0.0.1：34794有67397365760字节空闲;但我们预计需要减少输入184825113602 
 WARNING org.apache.hadoop.mapred.JobInProgress：没有空间来减少任务。节点tracker_worker2：localhost / 127.0.0.1：35984有67725717504字节空闲;但我们预计减少投入需要184825113602 
  

如何解决此问题？

请帮助解决方案

您的磁盘位于主机所在的减速器运行太小。您的减速机输入需要约。 172 gbs，但你只有63GB免费。

也许组合器可能对你有用。看看这里： http：//tumblr.mobocracy。 net / post / 6589143971 / using-a-combiner-with-hadoop-jobs

My map task completes successfully and I can see the application logs, but reducer stays in pending state

Kind    % Complete  Num Tasks   Pending Running Complete    Killed  Failed/Killed Task Attempts
map        100.00%  200 0   0   200 0   0 / 40
reduce  0.00%   1   1   0   0   0   0 / 0  

When I look at reduce task, I see

All Task Attempts

No Task Attempts found

When I see the hadoop-hduser-jobtracker-master.log, I see the following :

2011-10-31 00:00:00,238 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67730714624 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:01,634 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 68041449472 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:03,863 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397398528 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:04,532 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67729956864 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:04,695 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67998732288 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:06,865 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:08,057 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67948941312 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:09,566 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67728035840 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:11,118 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67907883008 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:12,258 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397365760 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:14,119 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67867385856 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:16,061 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:17,120 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67813683200 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:19,386 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67726835712 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:20,122 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67896156160 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:21,949 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:23,124 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 68150616064 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:25,078 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397365760 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:25,762 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67725717504 bytes free; but we expect reduce input to take 184825113602

How do I fix this issue?

Please help

解决方案

Your disk on the host where the reducer is run is too small. Your reducer input takes approx. 172 gbs, but you have only 63gb free.

Maybe combiner could be useful for you. Have a look here: http://tumblr.mobocracy.net/post/6589143971/using-a-combiner-with-hadoop-jobs

这篇关于hadoop-streaming：在待处理状态下减少任务，表示“没有空间来减少任务”。的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！