Hadoop 2.0名称节点，辅助节点和检查点节点以实现高可用性 [英] Hadoop 2.0 Name Node, Secondary Node and Checkpoint node for High Availability

问题描述

在阅读Apache Hadoop文档，对于理解二级节点的责任有一点点困惑。检查点节点

我在 Namenode 角色和责任上很清楚：

• NameNode将对文件系统的修改作为附加到本地文件系统文件的日志进行修改。当NameNode启动时，它从图像文件fsimage中读取HDFS状态，然后应用编辑日志文件中的编辑。然后它将新的HDFS状态写入fsimage，并使用空的编辑文件开始正常操作。由于NameNode仅在启动过程中合并fsimage和编辑文件，编辑日志文件可能会在繁忙集群上随时间变得非常大。更大的编辑文件的另一个副作用是，下次重新启动NameNode需要更长时间。

在理解Secondary namenode& amp; amp; amp;检查点namenode的责任。

辅助名称：

• 辅助NameNode定期合并fsimage和编辑日志文件，并将编辑日志大小保持在限制范围内。它通常在与主NameNode不同的机器上运行，因为它的内存需求与主NameNode的顺序相同。

检查点节点：

• 检查点节点定期创建名称空间的检查点。它从活动的NameNode下载fsimage和编辑，在本地合并它们，并将新图像上传回活动的NameNode。 Checkpoint节点通常运行在与NameNode不同的机器上，因为它的内存需求与NameNode的顺序相同。
  

p> 看起来，Secondary namenode&检查点节点不清楚。两者都在进行编辑。那么谁最终会修改呢？

另外，我在jira中创建了两个bug来消除理解这些概念时的不确定性。

  issues.apache.org/jira/browse/HDFS-8913 
 issues.apache.org/jira/browse / HDFS-8914 
  

解决方案NameNode（Primary）

NameNode存储HDFS的元数据。 HDFS的状态存储在一个名为fsimage的文件中，并且是元数据的基础。在运行期间，修改只会写入名为编辑的日志文件。在NameNode的下一次启动时，从fsimage读取状态，将编辑的更改应用于该状态，并将新状态写回fsimage。

检查点节点

$ b $在这些编辑被清除后，包含现在已准备好用于新的日志条目。 b

引入了Checkpoint节点来解决NameNode的缺点。这些更改只是写入编辑，并且在运行时不会合并到fsimage。如果NameNode运行一段时间，编辑变得非常庞大，下一次启动将花费更长的时间，因为必须对状态应用更多的更改以确定元数据的最后状态。

Checkpoint节点定期从NameNode获取fsimage和编辑并合并它们。结果状态称为检查点。在这之后，将结果上传到NameNode。

还有一种类似节点称为辅助节点，但它没有上传到NameNode特征。所以NameNode需要从Secondary NameNode获取状态。这也是因为名字暗示如果NameNode失败，Secondary NameNode接受请求，而不是这种情况。

备份节点

备份节点提供与检查点节点相同的功能，但与NameNode同步。它不需要定期获取更改，因为它收到一系列文件系统编辑。来自NameNode。它保存内存中的当前状态，只需将其保存到映像文件中即可创建新的检查点。

After reading Apache Hadoop documentation , there is a small confusion in understanding responsibilities of secondary node & check point node

I am clear on Namenode role and responsibilities:

• The NameNode stores modifications to the file system as a log appended to a native file system file, edits. When a NameNode starts up, it reads HDFS state from an image file, fsimage, and then applies edits from the edits log file. It then writes new HDFS state to the fsimage and starts normal operation with an empty edits file. Since NameNode merges fsimage and edits files only during start up, the edits log file could get very large over time on a busy cluster. Another side effect of a larger edits file is that next restart of NameNode takes longer.

But I have a small confusion in understanding Secondary namenode & Check point namenode responsibilities.

Secondary NameNode:

• The secondary NameNode merges the fsimage and the edits log files periodically and keeps edits log size within a limit. It is usually run on a different machine than the primary NameNode since its memory requirements are on the same order as the primary NameNode.

Check point node:

• The Checkpoint node periodically creates checkpoints of the namespace. It downloads fsimage and edits from the active NameNode, merges them locally, and uploads the new image back to the active NameNode. The Checkpoint node usually runs on a different machine than the NameNode since its memory requirements are on the same order as the NameNode. The Checkpoint node is started by bin/hdfs namenode -checkpoint on the node specified in the configuration file.

It seems that responsibility between Secondary namenode & Checkpoint node are not clear. Both are working on edits. So who will modify finally?

On a different note, I have created two bugs in jira to remove ambiguity in understanding these concepts.

issues.apache.org/jira/browse/HDFS-8913 
issues.apache.org/jira/browse/HDFS-8914 

解决方案

NameNode(Primary)

The NameNode stores the metadata of the HDFS. The state of HDFS is stored in a file called fsimage and is the base of the metadata. During the runtime modifications are just written to a log file called edits. On the next start-up of the NameNode the state is read from fsimage, the changes from edits are applied to that and the new state is written back to fsimage. After this edits is cleared and contains is now ready for new log entries.

Checkpoint Node

A Checkpoint Node was introduced to solve the drawbacks of the NameNode. The changes are just written to edits and not merged to fsimage during the runtime. If the NameNode runs for a while edits gets huge and the next startup will take even longer because more changes have to be applied to the state to determine the last state of the metadata.

The Checkpoint Node fetches periodically fsimage and edits from the NameNode and merges them. The resulting state is called checkpoint. After this is uploads the result to the NameNode.

There was also a similiar type of node called "Secondary Node" but it doesn’t have the "upload to NameNode" feature. So the NameNode need to fetch the state from the Secondary NameNode. It also was confussing because the name suggests that the Secondary NameNode takes the request if the NameNode fails which isn’t the case.

Backup Node

The Backup Node provides the same functionality as the Checkpoint Node, but is synchronized with the NameNode. It doesn’t need to fetch the changes periodically because it receives a strem of file system edits. from the NameNode. It holds the current state in-memory and just need to save this to an image file to create a new checkpoint.

这篇关于Hadoop 2.0名称节点，辅助节点和检查点节点以实现高可用性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！