以伪分布模式设置mapred.map.tasks时出错 [英] Error when setting mapred.map.tasks in pseudo-distributed mode

问题描述

正如建议此处，我我正在使用下面的mapred-site.xml文件以伪分布模式运行hadoop。该作业运行在4核心机器上。

 < configuration> 
<属性> 
<名称> mapred.job.tracker< / name> 
< value> localhost：9001< /值> 
< / property> 
<属性> 
<名称> mapred.map.tasks< / name> 
<值> 4< /值> 
< / property> 
<属性> 
<名称> mapred.reduce.tasks< / name> 
<值> 4< /值> 
< / property> 
 
< / configuration> 
  

我收到以下错误：

报告区块1.0000的比率已达到0.9990。

这是什么原因。如何解决这个问题？

解决方案

这不是问题，而是hadoop启动过程中的一个正常程序。 b

Safemode是HDFS状态，其中文件系统以只读方式挂载;不执行复制，也不能创建或删除文件。这会在NameNode启动时自动输入，以允许所有DataNode时间在NameNode中检入并在NameNode确定哪些块未被复制之前通知它们保存哪些块。

您也可以在此页面看到什么是Safemode NameNode一直等待，直到特定百分比的块出现并计入，该百分比由dfs.safemode.threshold.pct参数设置，该值在 code> hdfs-default.xml 默认情况下。但您可以在 hdfs-site.xml 中更改它。

它是

 <属性> 
<名称> dfs.safemode.threshold.pct< / name> 
<值> 0.999f< /值> 
< description> 
指定应满足
 dfs.replication.min定义的最小复制要求的块的百分比。 
小于或等于0的值表示在退出安全模式之前不等待块的任何特定
百分比。 
值大于1将使安全模式永久。 
< / description> 
< / property> 
  

所以你看到日志以0.9990结尾

 报告区块1.0000的比率已达到阈值0.9990。 
  

As suggested here, I am running hadoop in pseudodistributed mode with the following mapred-site.xml file. The job is running on a 4 core machine.

<configuration>
   <property>
        <name>mapred.job.tracker</name>
        <value>localhost:9001</value>  
   </property>
   <property>
     <name>mapred.map.tasks</name> 
     <value>4</value> 
  </property>
  <property>
     <name>mapred.reduce.tasks</name> 
     <value>4</value> 
  </property>

</configuration>

I am getting the following error:

The ratio of reported blocks 1.0000 has reached the threshold 0.9990.

What is the reason for this. How to fix this?

解决方案

It's not a problem, but a normal procedure during hadoop startup.

Safemode is an HDFS state in which the file system is mounted read-only; no replication is performed, nor can files be created or deleted. This is automatically entered as the NameNode starts, to allow all DataNodes time to check in with the NameNode and announce which blocks they hold, before the NameNode determines which blocks are under-replicated, etc.

You can also see what is Safemode in this page.

The NameNode waits until a specific percentage of the blocks are present and accounted-for, the percentage is set by dfs.safemode.threshold.pct parameter in hdfs-default.xml by default. But you can change it in hdfs-site.xml.

It's

<property>
  <name>dfs.safemode.threshold.pct</name>
  <value>0.999f</value>
  <description>
    Specifies the percentage of blocks that should satisfy 
    the minimal replication requirement defined by dfs.replication.min.
    Values less than or equal to 0 mean not to wait for any particular
    percentage of blocks before exiting safemode.
    Values greater than 1 will make safe mode permanent.
  </description>
</property>

So you see the log ends with 0.9990

The ratio of reported blocks 1.0000 has reached the threshold 0.9990.

这篇关于以伪分布模式设置mapred.map.tasks时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！