hadoop：任务不是本地文件？ [英] hadoop: tasks not local with file?
问题描述
我运行了一个hadoop工作，当我查看一些地图任务时，我发现他们没有运行文件块的位置。例如，地图任务在slave1上运行，但文件块（全部）在slave2中。这些文件都是gzip。
为什么会发生这种情况以及如何解决？
更新：请注意，有很多未完成的任务，所以这不是一个节点闲置的情况，因此可以托管从其他节点读取的任务。解决方案Hadoop的默认（FIFO）调度程序就像这样工作：当节点有足够的容量时，它会联系主服务器并询问为更多的工作。主服务器尝试分配数据本地任务或机架本地任务，但如果不能，它会将队列中的任何任务（等待任务）分配给该节点。然而，当这个节点被分配了这个非本地任务（我们称之为任务X）时，另一个节点有可能还有剩余容量并联系主请求工作。即使该节点实际上拥有X所需的本地数据副本，也不会分配该任务，因为其他节点能够比后者节点稍快地获得对主节点的锁定。这会导致数据位置较差，但FAST任务分配。 相比之下， Fair Scheduler 使用一种技术称为延迟时间表，通过延迟非本地访问来实现更高的地点任务分配一点点（可配置）。它实现了更高的地点，但延迟了一些任务的一小部分成本。
其他人正在研究更好的调度程序，这可能会得到改进在将来。现在，如果您希望获得更高的数据位置，您可以选择使用公平调度程序。
我不同意@ donald-miner的结论：使用默认复制因子3，你不会看到很多非数据本地的任务。他指出更多副本可以改善您的本地性％，但数据本地任务的百分比可能仍然很低是正确的。我也亲自跑过实验，看到FIFO调度器的数据位置非常低。如果你的工作很大（有很多任务），你可以实现高地位，但对于更普通，更小的工作，他们会遇到一个叫做头线调度的问题。从引用：
第一个局部性问题出现在小作业中（
具有小输入文件，因此只有少量数据
块要读取的作业） 。问题是，无论何时工作达到
排序列表的头部（即具有最少的
运行任务），其任务之一将在下一个插槽
上启动即变为空闲，不管该插槽位于哪个节点上。如果
的头部作业很小，则不太可能在
的数据上提供给它的节点。例如，一个有
10％节点数据的作业只能达到10％的位置。
引用Facebook生产集群中的数据，他们报告在大型生产环境中仅观察到5％的数据位置。
最后说明：如果您数据位置低？不是太多。 您工作的运行时间可能由散兵队员（需要更长时间才能完成的任务）和混洗阶段，因此仅改善数据位置在运行时间上有非常小的改善（如果有的话）。
I ran a hadoop job and when I look in some map tasks I see they are not running where the file's blocks are. E.g., the map task runs on slave1, but the file blocks (all of them) are in slave2. The files are all gzip.
Why is that happening and how to resolve?
UPDATE: note there are many pending tasks, so this is not a case of a node being idle and therefore hosting tasks that read from other nodes.
Hadoop's default (FIFO) scheduler works like this: When a node has spare capacity, it contacts the master and asks for more work. The master tries to assign a data-local task, or a rack-local task, but if it can't, it will assign any task in the queue (of waiting tasks) to that node. However, while this node was being assigned this non-local task (we'll call it task X), it is possible that another node also had spare capacity and contacted the master asking for work. Even if this node actually had a local copy of the data required by X, it will not be assigned that task because the other node was able to acquire the lock to the master slightly faster than the latter node. This results in poor data locality, but FAST task assignment.
In contrast, the Fair Scheduler uses a technique called delayed scheduling that achieves higher locality by delaying non-local task assignment for a "little bit" (configurable). It achieves higher locality but at a small cost of delaying some tasks.
Other people are working on better schedulers, and this may likely be improved in the future. For now, you can choose to use the Fair Scheduler if you wish to achieve higher data locality.
I disagree with @donald-miner's conclusion that "With a default replication factor of 3, you don't see very many tasks that are not data local." He is correct in noting that more replicas will give improve your locality %, but the percentage of data-local tasks may still be very low. I've also ran experiments myself and saw very low data locality with the FIFO scheduler. You could achieve high locality if your job is large (has many tasks), but for the more common, smaller jobs, they suffer from a problem called "head-of-line scheduling". Quoting from this paper:
The first locality problem occurs in small jobs (jobs that have small input files and hence have a small number of data blocks to read). The problem is that whenever a job reaches the head of the sorted list [...] (i.e. has the fewest running tasks), one of its tasks is launched on the next slot that becomes free, no matter which node this slot is on. If the head-of-line job is small, it is unlikely to have data on the node that is given to it. For example, a job with data on 10% of nodes will only achieve 10% locality.
That paper goes on to cite numbers from a production cluster at Facebook, and they reported observing just 5% of data locality in a large, production environment.
Final note: Should you care if you have low data locality? Not too much. The running time of your jobs may be dominated by the stragglers (tasks that take longer to complete) and shuffle phase, so improving data locality would only have a very modest improve in running time (if any at all).
这篇关于hadoop：任务不是本地文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
