Hadoop namenode需要在每台计算机启动后进行格式化 [英] Hadoop namenode needs to be formatted after every computer start

问题描述

我已经搜索过这个问题，虽然有很多类似的例子，但我找不到一个通用的解决方案或者适用于我的解决方案。我已经安装了Hadoop，并且正在伪分布式模式下运行。它工作正常，我可以多次启动和停止并使其正常运行。但是，如果我重新启动计算机并启动Hadoop，则namenode无法启动。我需要每次格式化，这意味着我失去了所有已完成的工作，需要重新开始。

我正在关注Hadoop：权威指南v3。

我的core-site.xml表示：

 < configuration> 
<属性> 
<名称> fs.default.name< /名称> 
< value> hdfs：// localhost /< / value> 
< / property> 
< / configuration> 
  

我的hdfs-site.xml显示：

 <结构> 
<属性> 
< name> dfs.replication< / name> 
<值> 1< /值> 
< / property> 
< / configuration> 
  

是否有一种配置Hadoop的方法，以便我不需要每次重新设置namenode的格式时间我重新启动电脑？

谢谢。

解决方案

您不覆盖hdfs配置dfs.name.dir，dfs.data.dir，默认情况下它指向/ tmp目录，当您的计算机重新启动时它将被清除。您必须通过覆盖位于HADOOP配置目录中的hdfs-site.xml文件中的这些值来将其从/ tmp更改为您主目录中的另一个位置。

以下步骤

在您的主目录中创建一个目录，以保存namenode image& datanode块（用您的登录名替换）

  mkdir / home /< USER> / pseudo / 
  code> 

修改您的HADOOP_CONF_DIR（hadoop配置目录）中的hdfs-site.xml文件，如下所示：



 <？xml version =1.0？> 
<？xml-stylesheet type =text / xslhref =configuration.xsl？> 
<配置> 
<属性> 
<名称> dfs.name.dir< /名称> 
< value> file：/// home /< USER> / pseudo / dfs / name< / value> 
< / property> 
<属性> 
<名称> dfs.data.dir< /名称> 
< value> file：/// home /< USER> / pseudo / dfs / data< / value> 
< / property> 
<属性> 
< name> dfs.replication< / name> 
<值> 1< /值> 
< / property> 
 
< / configuration> 
  

格式化您的hdfs namenode&开始使用


I have searched for this problem and while there are a number of similar examples I can't find a common solution or one that works for me. I have installed Hadoop and am running in pseudo distributed mode. It works fine, and I can start and stop it a number of times and get it running fine. However, if I re-start the computer and start Hadoop the namenode doesn't start. I need to format it every time, which means I lose all the work I have done and need to start again.

I am following Hadoop: The Definitive Guide v3.

My core-site.xml says:

<configuration>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost/</value>
    </property>
</configuration>

My hdfs-site.xml says:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

Is there a way of configuring Hadoop so that I don't need to re-format the namenode every time I restart the computer?

Thanks.

解决方案

Looks like you are not overriding the hdfs configurations dfs.name.dir , dfs.data.dir, by default it points to /tmp directory which will be cleared when your machine restarts. You have to change this from /tmp to another location in your home directory by overriding these values in your hdfs-site.xml file located in your HADOOP configuration directory.

Do the following steps

Create a directory in your home directory for keeping namenode image & datanode blocks (Replace with your login name)

mkdir /home/<USER>/pseudo/

Modify your hdfs-site.xml file in your HADOOP_CONF_DIR(hadoop configuration direcotry) as follows

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
  <name>dfs.name.dir</name>
  <value>file:///home/<USER>/pseudo/dfs/name</value>
</property>
<property>
  <name>dfs.data.dir</name>
  <value>file:///home/<USER>/pseudo/dfs/data</value>
</property>
<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>

</configuration>

Format your hdfs namenode & start using

这篇关于Hadoop namenode需要在每台计算机启动后进行格式化的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！