奇怪的 DNS 服务器导致 Hadoop 和 HBase 出现故障 [英] Weird DNS server causes Hadoop and HBase to malfunction

问题描述

我的网络有一些奇怪的(据我所知)DNS 服务器，这会导致 Hadoop 或 HBase 出现故障.

它将我的主机名解析为我的机器不知道的某个地址(即没有这样的接口).

如果我在/etc/hosts 中有以下条目，Hadoop 确实可以工作:

127.0.0.1 本地主机127.0.1.1 我的主机名

如果条目127.0.1.1 myhostname"不存在，则将文件上传到 HDFS 失败并抱怨它只能将文件复制到 0 个数据节点而不是 1 个.

但在这种情况下 HBase 不起作用:从 HBase shell 创建表会导致 NotAllMetaRegionsOnlineException(实际上是由 HMaster 试图绑定到 DNS 服务器为 myhostname 返回的错误地址引起的).

在其他网络中，我使用以下/etc/hosts:

127.0.0.1 本地主机192.168.1.1 我的主机名

Hadoop 和 HBase 都可以工作.问题是在第二个网络中，地址是动态的，我无法将其列出到/etc/hosts 中以覆盖由奇怪的 DNS 返回的结果.

Hadoop 以伪分布式模式运行.HBase 也运行在单节点上.

改变 DNS 服务器的行为不是一种选择.在 hbase/conf/regionservers 中将localhost"更改为 127.0.0.1 不会改变任何内容.

有人可以提出一种方法，我如何在保持互联网连接的同时覆盖其行为(我实际上通过 Teamviewer 在客户端的机器上工作).或者某种方式来配置 HBase(或它正在管理的 Zookeeper)不使用主机名来确定要绑定的地址?

解决方案

幸运的是，我找到了解决此 DNS 服务器问题的方法.

DNS 服务器在通过本地主机名查询时返回无效地址.HBase 默认对本地主机名进行反向 DNS 查找以确定绑定位置.由于DNS服务器返回的地址无效，HMaster无法绑定.

解决方法:在 hbase/conf/hbase-site.xml 中明确指定将用于 master 和 regionserver 的接口:

<预><代码><配置><财产><name>hbase.master.dns.interface</name><value>lo</value></属性><财产><name>hbase.regionserver.dns.interface</name><value>lo</value></属性></配置>

在本例中，我指定了用于主服务器和区域服务器的环回接口 (lo).

I have a network with some weird (as I understand) DNS server which causes Hadoop or HBase to malfunction.

It resolves my hostname to some address my machine doesn't know about (i.e. there is no such interface).

Hadoop does work if I have following entries in /etc/hosts:

127.0.0.1     localhost
127.0.1.1     myhostname

If entry "127.0.1.1 myhostname" is not present uploading file to HDFS fails and complains that it can replicate the file only to 0 datanodes instead of 1.

But in this case HBase does not work: creating a table from HBase shell causes NotAllMetaRegionsOnlineException (caused actually by HMaster trying to bind to wrong address returned by DNS server for myhostname).

In other network, I am using following /etc/hosts:

127.0.0.1   localhost
192.168.1.1 myhostname

And both Hadoop and HBase work. The problem is that in second network the address is dynamic and I can't list it into /etc/hosts to override result returned by weird DNS.

Hadoop is run in pseudo-distributed mode. HBase also runs on single node.

Changing behavior of DNS server is not an option. Changing "localhost" to 127.0.0.1 in hbase/conf/regionservers doesn't change anything.

Can somebody suggest a way how can I override its behavior while retaining internet connection (I actually work at client's machine through Teamviewer). Or some way to configure HBase (or Zookeeper it is managing) not to use hostname to determine address to bind?

解决方案

Luckily, I've found the workaround to this DNS server problem.

DNS server returned invalid address when queried by local hostname. HBase by default does reverse DNS lookup on local hostname to determine where to bind. Because the address returned by DNS server was invalid, HMaster wasn't able to bind.

Workaround: In hbase/conf/hbase-site.xml explicitly specify interfaces that will be used for master and regionserver:

<configuration>
  <property>
    <name>hbase.master.dns.interface</name>
    <value>lo</value>
  </property>
  <property>
    <name>hbase.regionserver.dns.interface</name>
    <value>lo</value>
  </property>
</configuration>

In this case, I specified loopback interface (lo) to be used for both master and regionserver.

这篇关于奇怪的 DNS 服务器导致 Hadoop 和 HBase 出现故障的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！