Cloudera的：在HDFS异常上传文件 [英] Cloudera: upload a File in the HDFS Exception
本文介绍了Cloudera的：在HDFS异常上传文件的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
我使用的是Mac OS X的优胜美地与VM Cloudera的-快速入门-VM-5.4.2-0-VirtualBox的。当我输入HDFS DFS -put TESTFILE.TXT把一个文本文件导入HDFS我得到的 DataStreamer异常。我注意到，主要的问题是，我有节点的数目是零。我复制到这里完整的错误消息下面，我想知道我应该做的解决这个问题。
＆GT; WARN hdfs.DFSClient：DataStreamer
＆GT; Exceptionorg.apache.hadoop.ipc.RemoteException（java.io.IOException异常）：
＆GT;文件/user/cloudera/testfile.txt._COPYING_只能被复制到
＆GT; 0的节点，而不是minReplication（= 1）。有0数据节点（S）
＆GT;运行，并且没有节点（多个）被排除在此操作。在
＆GT; org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1541)
＆GT;在
＆GT; org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock（FSNamesystem.java:3286）
＆GT;在
＆GT; org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock（NameNodeRpcServer.java:667）
＆GT;在
＆GT; org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:212)
＆GT;在
＆GT; org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:483)
＆GT;在
＆GT; org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
＆GT;在
＆GT; org.apache.hadoop.ipc.ProtobufRpcEngine $服务器$ ProtoBufRpcInvoker.call（ProtobufRpcEngine.java:619）
＆GT;在org.apache.hadoop.ipc.RPC $ Server.call（RPC.java:1060）在
＆GT; org.apache.hadoop.ipc.Server $ $处理程序1.run（Server.java:2044）在
＆GT; org.apache.hadoop.ipc.Server $ $处理程序1.run（Server.java:2040）在
＆GT; java.security.AccessController.doPrivileged（本机方法）的
＆GT; javax.security.auth.Subject.doAs（Subject.java:415）在
＆GT; org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1671）
＆GT;在org.apache.hadoop.ipc.Server $ Handler.run（Server.java:2038）在
＆GT; org.apache.hadoop.ipc.Client.call（Client.java:1468）在
＆GT; org.apache.hadoop.ipc.Client.call（Client.java:1399）在
＆GT; org.apache.hadoop.ipc.ProtobufRpcEngine $ Invoker.invoke（ProtobufRpcEngine.java:232）
＆GT;在com.sun.proxy。$ Proxy14.addBlock（来源不明）在
＆GT; org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
＆GT;在sun.reflect.NativeMethodAccessorImpl.invoke0（本机方法）的
＆GT; sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:57）
＆GT;在
＆GT; sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）
＆GT;在java.lang.reflect.Method.invoke（Method.java:606）在
＆GT; org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod（RetryInvocationHandler.java:187）
＆GT;在
＆GT; org.apache.hadoop.io.retry.RetryInvocationHandler.invoke（RetryInvocationHandler.java:102）
＆GT;在com.sun.proxy。$ Proxy15.addBlock（来源不明）在
＆GT; org.apache.hadoop.hdfs.DFSOutputStream $ DataStreamer.locateFollowingBlock（DFSOutputStream.java:1544）
＆GT;在
＆GT; org.apache.hadoop.hdfs.DFSOutputStream $ DataStreamer.nextBlockOutputStream（DFSOutputStream.java:1361）
＆GT;在
＆GT; org.apache.hadoop.hdfs.DFSOutputStream $ DataStreamer.run（DFSOutputStream.java:600）提出：
＆GT;文件/user/cloudera/testfile.txt._COPYING_只能被复制到
＆GT; 0的节点，而不是minReplication（= 1）。有0数据节点（S）
＆GT;运行，并且没有节点（多个）被排除在本
＆GT;操作。[Cloudera的@快速入门〜] $ HDFS DFS -put testfile.txt15 / 10月18日
＆GT;三点51分51秒WARN hdfs.DFSClient：DataStreamer
＆GT; Exceptionorg.apache.hadoop.ipc.RemoteException（java.io.IOException异常）：
＆GT;文件/user/cloudera/testfile.txt._COPYING_只能被复制到
＆GT; 0的节点，而不是minReplication（= 1）。有0数据节点（S）
＆GT;运行，并且没有节点（多个）被排除在此操作。在
＆GT; org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1541)
＆GT;在
＆GT; org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock（FSNamesystem.java:3286）
＆GT;在
＆GT; org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock（NameNodeRpcServer.java:667）
＆GT;在
＆GT; org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:212)
＆GT;在
＆GT; org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:483)
＆GT;在
＆GT; org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
＆GT;在
＆GT; org.apache.hadoop.ipc.ProtobufRpcEngine $服务器$ ProtoBufRpcInvoker.call（ProtobufRpcEngine.java:619）
＆GT;在org.apache.hadoop.ipc.RPC $ Server.call（RPC.java:1060）在
＆GT; org.apache.hadoop.ipc.Server $ $处理程序1.run（Server.java:2044）在
＆GT; org.apache.hadoop.ipc.Server $ $处理程序1.run（Server.java:2040）在
＆GT; java.security.AccessController.doPrivileged（本机方法）的
＆GT; javax.security.auth.Subject.doAs（Subject.java:415）在
＆GT; org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1671）
＆GT;在org.apache.hadoop.ipc.Server $ Handler.run（Server.java:2038）在
＆GT; org.apache.hadoop.ipc.Client.call（Client.java:1468）在
＆GT; org.apache.hadoop.ipc.Client.call（Client.java:1399）在
＆GT; org.apache.hadoop.ipc.ProtobufRpcEngine $ Invoker.invoke（ProtobufRpcEngine.java:232）
＆GT;在com.sun.proxy。$ Proxy14.addBlock（来源不明）在
＆GT; org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
＆GT;在sun.reflect.NativeMethodAccessorImpl.invoke0（本机方法）的
＆GT; sun.reflect.NativeMethodAccessorImpl.invoke（NativeMethodAccessorImpl.java:57）
＆GT;在
＆GT; sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）
＆GT;在java.lang.reflect.Method.invoke（Method.java:606）在
＆GT; org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod（RetryInvocationHandler.java:187）
＆GT;在
＆GT; org.apache.hadoop.io.retry.RetryInvocationHandler.invoke（RetryInvocationHandler.java:102）
＆GT;在com.sun.proxy。$ Proxy15.addBlock（来源不明）在
＆GT; org.apache.hadoop.hdfs.DFSOutputStream $ DataStreamer.locateFollowingBlock（DFSOutputStream.java:1544）
＆GT;在
＆GT; org.apache.hadoop.hdfs.DFSOutputStream $ DataStreamer.nextBlockOutputStream（DFSOutputStream.java:1361）
＆GT;在
＆GT; org.apache.hadoop.hdfs.DFSOutputStream $ DataStreamer.run（DFSOutputStream.java:600）提出：
＆GT;文件/user/cloudera/testfile.txt._COPYING_只能被复制到
＆GT; 0的节点，而不是minReplication（= 1）。有0数据节点（S）
＆GT;运行，并且没有节点（多个）被排除在本
＆GT;操作。[Cloudera的@快速入门〜] $
解决方案
1。停止Hadoop的服务，在 <一个描述href=\"http://www.cloudera.com/content/www/en-us/documentation/archive/cdh/4-x/4-7-1/CDH4-Installation-Guide/cdh4ig_topic_27_3.html\"相对=nofollow>停止服务
在`CD /etc/init.d中X; LS的Hadoop *`;做须藤服务$ X停止; DONE
2。卸下 的/ var / lib中/ Hadoop的HDFS /缓存/
须藤RM -r在/ var / lib中/ Hadoop的HDFS /缓存/
3。格式的Namenode
sudo的-u HDFS HDFS的NameNode -format注意：答案用大写字母ÿ注：数据格式在过程中丢失。
4。启动Hadoop的服务
在`CD /etc/init.d中X; LS的Hadoop *`;做须藤服务$ X启动; DONE
5。确保您的系统是不低的磁盘空间运行。您还可以确认这一点，如果有关于在日志文件中的低磁盘空间的任何警告。
6。创建/ tmp目录
删除旧的/ tmp目录，如果它存在：
$ sudo的-u Hadoop的HDFS FS -rm -r / tmp目录创建一个新的/ tmp目录并设置权限： $ sudo的-u Hadoop的HDFS FS -mkdir / tmp目录
$ sudo的-u Hadoop的HDFS FS -chmod -R 1777 / tmp目录
7。创建用户目录：
$ sudo的-u Hadoop的HDFS FS -mkdir /用户/＆lt;使用者＆GT;
$ sudo的-u Hadoop的HDFS FS -chown＆lt;使用者＆GT; /用户/＆lt;使用者＆GT;其中，＆lt;使用者名称＆gt;是Linux用户名
I use a MAC OS X Yosemite with a VM cloudera-quickstart-vm-5.4.2-0-virtualbox. When I type "hdfs dfs -put testfile.txt" to put a TEXT FILE into HDFS I get a DataStreamer Exception. I notice that the main problem is that the number of nodes that I have is null. I copy here below the complete error message and I would like to know how should I do to solve this.
> WARN hdfs.DFSClient: DataStreamer
> Exceptionorg.apache.hadoop.ipc.RemoteException(java.io.IOException):
> File /user/cloudera/testfile.txt._COPYING_ could only be replicated to
> 0 nodes instead of minReplication (=1). There are 0 datanode(s)
> running and no node(s) are excluded in this operation. at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1541)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3286)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:667)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:212)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:483)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at
> org.apache.hadoop.ipc.Client.call(Client.java:1468) at
> org.apache.hadoop.ipc.Client.call(Client.java:1399) at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy14.addBlock(Unknown Source) at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy15.addBlock(Unknown Source) at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600)put:
> File /user/cloudera/testfile.txt._COPYING_ could only be replicated to
> 0 nodes instead of minReplication (=1). There are 0 datanode(s)
> running and no node(s) are excluded in this
> operation.[cloudera@quickstart ~]$ hdfs dfs -put testfile.txt15/10/18
> 03:51:51 WARN hdfs.DFSClient: DataStreamer
> Exceptionorg.apache.hadoop.ipc.RemoteException(java.io.IOException):
> File /user/cloudera/testfile.txt._COPYING_ could only be replicated to
> 0 nodes instead of minReplication (=1). There are 0 datanode(s)
> running and no node(s) are excluded in this operation. at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1541)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3286)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:667)
> at
> org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:212)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:483)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:415) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at
> org.apache.hadoop.ipc.Client.call(Client.java:1468) at
> org.apache.hadoop.ipc.Client.call(Client.java:1399) at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy14.addBlock(Unknown Source) at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606) at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy15.addBlock(Unknown Source) at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600)put:
> File /user/cloudera/testfile.txt._COPYING_ could only be replicated to
> 0 nodes instead of minReplication (=1). There are 0 datanode(s)
> running and no node(s) are excluded in this
> operation.[cloudera@quickstart ~]$
解决方案
1. Stop Hadoop services as describe in Stopping Services
for x in `cd /etc/init.d ; ls hadoop*` ; do sudo service $x stop ; done
2. Remove all files from /var/lib/hadoop-hdfs/cache/
sudo rm -r /var/lib/hadoop-hdfs/cache/
3. Format Namenode
sudo -u hdfs hdfs namenode -format
Note: Answer with a capital Y
Note: Data is lost during format process.
4. Start Hadoop Services
for x in `cd /etc/init.d ; ls hadoop*` ; do sudo service $x start ; done
5. Ensure that your system is not running on low diskspace. You can also confirm this if there is any WARNING about low diskspace in Log files.
6. Create the /tmp Directory
Remove the old /tmp if it exists:
$ sudo -u hdfs hadoop fs -rm -r /tmp
Create a new /tmp directory and set permissions:
$ sudo -u hdfs hadoop fs -mkdir /tmp
$ sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
7. Create User Directories:
$ sudo -u hdfs hadoop fs -mkdir /user/<user>
$ sudo -u hdfs hadoop fs -chown <user> /user/<user>
where <user> is the Linux username
这篇关于Cloudera的：在HDFS异常上传文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
