如何将文件放到特定节点? [英] How to put files to specific node?

问题描述

是否可以告诉 HDFS 在哪里存储特定文件?

Is it possible to tell HDFS where to store particular files?

用例

我刚刚将第 1 批文件加载到 HDFS 中，并且想要对这些数据运行作业/应用程序.但是，我还有第 2 批仍待加载.如果我可以在第一批上运行作业/应用程序，比如从 1 到 10 的节点，并将新数据加载到节点，比如 11 到 20，完全并行，那就太好了.

I've just loaded batch #1 of files into HDFS and want to run job/application on these data. However, I also have batch #2 that is still to be loaded. It would be nice if I could run job/application on first batch on, say, nodes from 1 to 10, and load new data to nodes, say, 11 to 20, completely in parallel.

最初我认为 NameNode federation (Hadoop 2.x) 正是这样做的，但看起来联邦只拆分了命名空间，而 DataNode 仍然为所有连接的 NameNode 提供块.

Initially I thought that NameNode federation (Hadoop 2.x) does exactly that, but it looks like federation only splits namespace, while DataNodes still provide blocks for all connected NameNodes.

那么，有没有办法在 HDFS 中控制数据的分布?它真的有意义吗?

So, is there a way to control the distribution of data in HDFS? And does it make sense at all?

推荐答案

从技术上讲，您可以，但我不会.

Technically, you can, but I wouldn't.

如果您想完全控制数据的去向，您可以扩展 BlockPlacementPolicy(参见 hdfs 如何选择要存储的数据节点).这并不容易，我不推荐这样做.

If you want full control over where the data goes, you can extend BlockPlacementPolicy (see how does hdfs choose a datanode to store). This won't be easy to do and I don't recommend it.

您可以采取一些措施，通过一些巧妙的设置来最大限度地减少两组节点之间的流量，以利用机架感知来发挥您的优势.

You can probably take steps to minimize the amount of traffic between your two sets of nodes with some clever setup to use rack-awareness to your advantage.

这篇关于如何将文件放到特定节点?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！