Hadoop namenode：单点故障 [英] Hadoop namenode : Single point of failure

问题描述

Hadoop架构中的Namenode是单点故障。

拥有大型Hadoop集群的人如何应对这个问题？

是否有行业认可的解决方案能够很好地工作，其中一个次要的名称节点接管主要节点失败的情况？ 解决方案Yahoo在特定推荐中针对不同群集大小的配置设置采取了NameNode失败考虑到。例如：

Hadoop集群中的单点故障是NameNode。虽然任何其他机器（间歇或永久）的丢失不会导致数据丢失，但NameNode丢失会导致群集不可用。 NameNode数据的永久丢失会导致集群的HDFS无法运行。

因此，在此配置中应该采取另一步骤来备份NameNode元数据。 $ b

Facebook使用 Hadoop的微调版本为其数据仓库;它有一些优化，重点关注NameNode的可靠性。除了在github上提供的补丁，Facebook似乎专门用于快速使用 AvatarNode 在主要和次要NameNode之间切换。 Dhruba Borthakur的博客包含其他几个条目，提供了有关NameNode作为单点故障的进一步见解。

编辑：有关Facebook对NameNode的改进的更多信息。

The Namenode in the Hadoop architecture is a single point of failure.

How do people who have large Hadoop clusters cope with this problem?.

Is there an industry-accepted solution that has worked well wherein a secondary Namenode takes over in case the primary one fails ?

解决方案

Yahoo has certain recommendations for configuration settings at different cluster sizes to take NameNode failure into account. For example:

The single point of failure in a Hadoop cluster is the NameNode. While the loss of any other machine (intermittently or permanently) does not result in data loss, NameNode loss results in cluster unavailability. The permanent loss of NameNode data would render the cluster's HDFS inoperable.

Therefore, another step should be taken in this configuration to back up the NameNode metadata

Facebook uses a tweaked version of Hadoop for its data warehouses; it has some optimizations that focus on NameNode reliability. Additionally to the patches available on github, Facebook appears to use AvatarNode specifically for quickly switching between primary and secondary NameNodes. Dhruba Borthakur's blog contains several other entries offering further insights into the NameNode as a single point of failure.

Edit: Further info about Facebook's improvements to the NameNode.

这篇关于Hadoop namenode：单点故障的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！