hdfs dfs -ls命令的结果 [英] Result of hdfs dfs -ls command
问题描述
在执行hdfs dfs -ls命令时，我想知道结果是存储在群集中的所有文件，还是只是执行它的节点中的分区。
我是hadoop中的一个newby，我在每个节点中都会遇到一些问题。
谢谢
问题：...如果结果是存储在集群中的所有文件或......
您从 ls 命令中看到的是集群中存储的所有文件。更具体地说，你看到的是一堆文件路径和名称。这些信息是命名空间的一部分，由Namenode管理。
...只是执行节点的分区如果你认为hdfs在这个节点上保留了一些文件，而在另一个节点上保留了一些文件。你误会了。没有这样的事情。 NameNode记录名称空间和blocksMap。实际上，文件由块组成。 NameNode知道文件具有多少个块，并且在哪些DataNode上保留块。 NameNode决定块的保存位置，它对用户是透明的。每个块默认有3个复制，每个复制都在一个DataNode上。所以假设一个文件有2个块，它最多可以位于6个DataNode上，没有DataNode保存完整的文件（在本例中为true），因为在另一种常见的情况下，当一个小文件只有1个块时，每个复制都是一个完整的文件）。
有关更多信息，请查看官方文档 Hdfs Design
In the execution of hdfs dfs -ls command I wuold like to know if the result are all the files stored in the cluster or just the partitions in the node where it is executed. I'm a newby in hadoop and I´m having some problems serching the partitions in each node.
Thank you
解决方案Question: "...if the result are all the files stored in the cluster or..."
What you see from
lscommand are all the files stored in the cluster. More specifically, what you see is a bunch of file paths and names. These information is part of namespace, which is managed by a Namenode."...just the partitions in the node where it is executed.."
If you thought hdfs keeps some files on this node, and some files on the other node. You misunderstood. There's no such thing. NameNode keeps tracks of namespace, and blocksMap. In fact, Files are composed of blocks. NameNode knows the file has how many blocks and on which DataNodes the blocks are kept. NameNode decides where the blocks are kept, it's transparent to the user. Each block has 3 replication by default, and each replication is on one DataNode. So Assume a file has 2 blocks, it could be located on at most 6 DataNodes, No DataNode keeps the complete files(true in this example. Because in another common case when a small file has only 1 block, each replication is a complete file).
For more information, take a look at the official document of Hdfs Design
这篇关于hdfs dfs -ls命令的结果的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
