连接并持久化到 HBase [英] Connecting and Persisting to HBase
本文介绍了连接并持久化到 HBase的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我只是尝试使用 java 客户端连接到 hbase，它是 cloudera-vm 的一部分.
(192.168.56.102 是虚拟机的 inet ip)
我使用只有主机网络设置的虚拟机.
这样我就可以访问hbase master的webUI@http://192.168.56.102:60010/master.jsp
我的 java 客户端(在 vm 本身上运行良好)也建立了到 192.168.56.102:2181 的连接
但是当它调用 getMaster 时，我得到 连接被拒绝 查看日志:
11/09/14 11:19:30 INFO zookeeper.ZooKeeper:发起客户端连接，connectString=192.168.56.102:2181 sessionTimeout=180000 watcher=hconnection11/09/14 11:19:30 INFO zookeeper.ClientCnxn:打开到服务器的套接字连接/192.168.56.102:218111/09/14 11:19:30 INFO zookeeper.ClientCnxn:已建立到 cloudera-vm/192.168.56.102:2181 的套接字连接，启动会话11/09/14 11:19:30 INFO zookeeper.ClientCnxn:在服务器 cloudera-vm/192.168.56.102:2181 上完成会话建立，sessionid = 0x13267157f930009，协商超时 = 4000011/09/14 11:19:32 INFO client.HConnectionManager$HConnectionImplementation:getMaster 尝试 0 of 10 失败；睡眠 1000 次后重试java.net.ConnectException:连接被拒绝:没有更多信息在 sun.nio.ch.SocketChannelImpl.checkConnect(本机方法)在 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)在 org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)在 org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)在 org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)在 org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)在 org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)在 org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)在 $Proxy9.getProtocolVersion(来源不明)在 org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)在 org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)在 org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)在 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:358)在 org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:90)在 org.datanucleus.store.hbase.HBaseUtils$1.run(HBaseUtils.java:432)在 java.security.AccessController.doPrivileged(Native Method)在 org.datanucleus.store.hbase.HBaseUtils.createSchemaForClass(HBaseUtils.java:428)在 org.datanucleus.store.hbase.HBaseMetaDataListener.loaded(HBaseMetaDataListener.java:74)在 org.datanucleus.store.hbase.HBaseStoreManager.(HBaseStoreManager.java:76)在 sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)在 sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)在 sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)在 java.lang.reflect.Constructor.newInstance(Constructor.java:513)在 org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:677)在 org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:287)在 org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:453)在 org.datanucleus.NucleusContext.initialise(NucleusContext.java:264)在 org.datanucleus.api.jpa.JPAEntityManagerFactory.initialiseNucleusContext(JPAEntityManagerFactory.java:746)在 org.datanucleus.api.jpa.JPAEntityManagerFactory.(JPAEntityManagerFactory.java:422)在 org.datanucleus.api.jpa.PersistenceProviderImpl.createEntityManagerFactory(PersistenceProviderImpl.java:91)在 javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:150)在 javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:70)在 de.syrtec.jpa2hbase.start.TestDAO.main(TestDAO.java:13) hbase-site.xml:
<预><代码><配置><财产><name>hbase.zookeeper.quorum</name><值>192.168.56.102</值></属性><财产><name>hbase.zookeeper.property.clientPort</name><value>2181</value></属性><财产><name>hbase.master</name><值>192.168.56.102:60010</值></属性></配置>持久性.xml:
<持久化xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd"版本=1.0"><persistence-unit name="hbase-addressbook"交易类型="RESOURCE_LOCAL"><属性><property name="datanucleus.ConnectionURL" value="hbase:192.168.56.102:60010"/><property name="datanucleus.ConnectionUserName" value=""/><property name="datanucleus.ConnectionPassword" value=""/><property name="datanucleus.autoCreateSchema" value="true"/><property name="datanucleus.validateTables" value="false"/><property name="datanucleus.Optimistic" value="false"/><property name="datanucleus.validateConstraints" value="false"/></属性></persistence-unit></持久性>TestDAO.java:
公共类 TestDAO {公共静态无效主(字符串 [] args){EntityManagerFactory emf = Persistence.createEntityManagerFactory("hbase-addressbook");EntityManager em = emf.createEntityManager();实体交易 tx = null;Account a2 = new Account("myPre3", "mySur3", 2, new Login("e", "f"));tx = em.getTransaction();tx.begin();em.persist(a2);tx.commit();}}只是在 VM 上运行的网络服务的添加 (netstat -ntpl):
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 563/sshdtcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 909/cupsdtcp 0 0 0.0.0.0:8088 0.0.0.0:* 听 2004/python2.6tcp 0 0 127.0.0.1:8001 0.0.0.0:* 听 2345/python2.6tcp6 0 0 :::39087 :::* 听 1327/javatcp6 0 0 :::54162 :::* 听 1411/javatcp6 0 0 127.0.0.1:8020 :::* 听 1411/javatcp6 0 0 127.0.0.1:8021 :::* 听 1327/javatcp6 0 0 :::50070 :::* 听 1411/javatcp6 0 0 :::22 :::* LISTEN 563/sshdtcp6 0 0 ::1:631 :::* LISTEN 909/cupsdtcp6 0 0 :::11000 :::* 听 2077/javatcp6 0 0 :::43738 :::* 听 858/javatcp6 0 0 :::50010 :::* 听 858/javatcp6 0 0 :::51258 :::* 听 858/javatcp6 0 0 :::50075 :::* 听 858/javatcp6 0 0 :::60030 :::* 听 1726/javatcp6 0 0 127.0.0.1:44446 :::* 听 1642/javatcp6 0 0 127.0.1.1:41246 :::* 听 1726/javatcp6 0 0 :::8002 :::* LISTEN 2009/javatcp6 0 0 :::8003 :::* LISTEN 2009/javatcp6 0 0 127.0.1.1:58915 ::* 听 1726/javatcp6 0 0 :::59203 :::* 听 1491/javatcp6 0 0 :::50020 :::* 听 858/javatcp6 0 0 127.0.0.1:8005 :::* 听 2077/javatcp6 0 0 :::2181 :::* 听 1726/javatcp6 0 0 :::9290 :::* 听 1327/javatcp6 0 0 :::60010 :::* 听 1726/javatcp6 0 0 :::50090 :::* 听 1491/javatcp6 0 0 :::10090 :::* 听 1411/javatcp6 0 0 :::50060 :::* 听 1642/javatcp6 0 0 :::50030 :::* 听 1327/java 解决方案
经过多天乏味的研究，我可以提供一个解决方案..可以让其他人免于很多麻烦..
问题的根源在于 Ubuntu 的 IPV6 支持(我在这里使用).但仅仅在网络管理中停用 IPV6 是不够的.
相反，您必须将以下内容附加到文件中:/etc/sysctl.conf
#禁用ipv6net.ipv6.conf.all.disable_ipv6 = 1net.ipv6.conf.default.disable_ipv6 = 1net.ipv6.conf.lo.disable_ipv6 = 1重启后，您应该通过以下方式验证 IPV6 是否真的关闭:
cat/proc/sys/net/ipv6/conf/all/disable_ipv6(0 = IPV6 开启；1 = IPV6 关闭)
非常感谢
这给了我关键的提示.
I just tried to connect to hbase which is part of the cloudera-vm using a java client.
(192.168.56.102 is the inet ip of the vm)
I use virtual box with host only network setting.
So I can access the webUI of the hbase master @ http://192.168.56.102:60010/master.jsp
Also my java client (worked well on the vm itself) established connection to 192.168.56.102:2181
But when it calls getMaster I get connection refused see log:
11/09/14 11:19:30 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=192.168.56.102:2181 sessionTimeout=180000 watcher=hconnection
11/09/14 11:19:30 INFO zookeeper.ClientCnxn: Opening socket connection to server /192.168.56.102:2181
11/09/14 11:19:30 INFO zookeeper.ClientCnxn: Socket connection established to cloudera-vm/192.168.56.102:2181, initiating session
11/09/14 11:19:30 INFO zookeeper.ClientCnxn: Session establishment complete on server cloudera-vm/192.168.56.102:2181, sessionid = 0x13267157f930009, negotiated timeout = 40000
11/09/14 11:19:32 INFO client.HConnectionManager$HConnectionImplementation: getMaster attempt 0 of 10 failed; retrying after sleep of 1000
java.net.ConnectException: Connection refused: no further information
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:311)
at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:865)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:732)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy9.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getMaster(HConnectionManager.java:358)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
at org.datanucleus.store.hbase.HBaseUtils$1.run(HBaseUtils.java:432)
at java.security.AccessController.doPrivileged(Native Method)
at org.datanucleus.store.hbase.HBaseUtils.createSchemaForClass(HBaseUtils.java:428)
at org.datanucleus.store.hbase.HBaseMetaDataListener.loaded(HBaseMetaDataListener.java:74)
at org.datanucleus.store.hbase.HBaseStoreManager.<init>(HBaseStoreManager.java:76)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:677)
at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:287)
at org.datanucleus.NucleusContext.createStoreManagerForProperties(NucleusContext.java:453)
at org.datanucleus.NucleusContext.initialise(NucleusContext.java:264)
at org.datanucleus.api.jpa.JPAEntityManagerFactory.initialiseNucleusContext(JPAEntityManagerFactory.java:746)
at org.datanucleus.api.jpa.JPAEntityManagerFactory.<init>(JPAEntityManagerFactory.java:422)
at org.datanucleus.api.jpa.PersistenceProviderImpl.createEntityManagerFactory(PersistenceProviderImpl.java:91)
at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:150)
at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:70)
at de.syrtec.jpa2hbase.start.TestDAO.main(TestDAO.java:13)
hbase-site.xml:
<configuration>
<property>
<name>hbase.zookeeper.quorum</name>
<value>192.168.56.102</value>
</property>
<property>
<name>hbase.zookeeper.property.clientPort</name>
<value>2181</value>
</property>
<property>
<name>hbase.master</name>
<value>192.168.56.102:60010</value>
</property>
</configuration>
persistence.xml:
<persistence
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_1_0.xsd"
version="1.0">
<persistence-unit name="hbase-addressbook"
transaction-type="RESOURCE_LOCAL">
<properties>
<property name="datanucleus.ConnectionURL" value="hbase:192.168.56.102:60010" />
<property name="datanucleus.ConnectionUserName" value="" />
<property name="datanucleus.ConnectionPassword" value="" />
<property name="datanucleus.autoCreateSchema" value="true" />
<property name="datanucleus.validateTables" value="false" />
<property name="datanucleus.Optimistic" value="false" />
<property name="datanucleus.validateConstraints" value="false" />
</properties>
</persistence-unit>
</persistence>
TestDAO.java:
public class TestDAO {
public static void main(String[] args) {
EntityManagerFactory emf = Persistence.createEntityManagerFactory("hbase-addressbook");
EntityManager em = emf.createEntityManager();
EntityTransaction tx = null;
Account a2 = new Account("myPre3", "mySur3", 2, new Login("e", "f"));
tx = em.getTransaction();
tx.begin();
em.persist(a2);
tx.commit();
}
}
just an addition of the network services running on the VM (netstat -ntpl):
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 563/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 909/cupsd
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN 2004/python2.6
tcp 0 0 127.0.0.1:8001 0.0.0.0:* LISTEN 2345/python2.6
tcp6 0 0 :::39087 :::* LISTEN 1327/java
tcp6 0 0 :::54162 :::* LISTEN 1411/java
tcp6 0 0 127.0.0.1:8020 :::* LISTEN 1411/java
tcp6 0 0 127.0.0.1:8021 :::* LISTEN 1327/java
tcp6 0 0 :::50070 :::* LISTEN 1411/java
tcp6 0 0 :::22 :::* LISTEN 563/sshd
tcp6 0 0 ::1:631 :::* LISTEN 909/cupsd
tcp6 0 0 :::11000 :::* LISTEN 2077/java
tcp6 0 0 :::43738 :::* LISTEN 858/java
tcp6 0 0 :::50010 :::* LISTEN 858/java
tcp6 0 0 :::51258 :::* LISTEN 858/java
tcp6 0 0 :::50075 :::* LISTEN 858/java
tcp6 0 0 :::60030 :::* LISTEN 1726/java
tcp6 0 0 127.0.0.1:44446 :::* LISTEN 1642/java
tcp6 0 0 127.0.1.1:41246 :::* LISTEN 1726/java
tcp6 0 0 :::8002 :::* LISTEN 2009/java
tcp6 0 0 :::8003 :::* LISTEN 2009/java
tcp6 0 0 127.0.1.1:58915 :::* LISTEN 1726/java
tcp6 0 0 :::59203 :::* LISTEN 1491/java
tcp6 0 0 :::50020 :::* LISTEN 858/java
tcp6 0 0 127.0.0.1:8005 :::* LISTEN 2077/java
tcp6 0 0 :::2181 :::* LISTEN 1726/java
tcp6 0 0 :::9290 :::* LISTEN 1327/java
tcp6 0 0 :::60010 :::* LISTEN 1726/java
tcp6 0 0 :::50090 :::* LISTEN 1491/java
tcp6 0 0 :::10090 :::* LISTEN 1411/java
tcp6 0 0 :::50060 :::* LISTEN 1642/java
tcp6 0 0 :::50030 :::* LISTEN 1327/java
解决方案
After many tedious days of research I can provide a solution..could save others from a lot of headaches..
The root of the problem is the IPV6 support of Ubuntu (I'm using here). But just deactivating IPV6 in the network administration wasn't enough.
Instead you have to append the following to the file: /etc/sysctl.conf
#disable ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
After reboot you should validate that IPV6 is really off by:
cat /proc/sys/net/ipv6/conf/all/disable_ipv6
(0 = IPV6 on ; 1 = IPV6 off)
Thanks a lot to
which gave me the crucial hint.
这篇关于连接并持久化到 HBase的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
