“超级模式”的目的是什么？在hadoop？ [英] What is the purpose of "uber mode" in hadoop?

问题描述

我是一个大数据新手。我在互联网上搜索了很多超级模式。我越搜索越多，我感到困惑。任何人都可以通过回答我的问题来帮助我吗？

• 超级模式有什么作用？


• 它在mapred 1.x和2.x中的工作方式不同吗？


• 我可以在哪里找到它的设置？

解决方案

$ b $在Hadoop2中的UBER模式是什么？ b

通常，映射器和缩减器将由ResourceManager（RM）运行，RM将为映射器和缩减器创建单独的容器。
优步配置，将允许在与ApplicationMaster（AM）相同的流程中运行映射器和缩减器。

优步工作：

$ b 优步工作是在MapReduce ApplicationMaster。而是与RM通信以创建映射器和简化器容器。
AM在自己的进程中运行地图并减少任务，并避免启动和与远程容器通信的开销。

为什么 b
$ b

如果你有一个小数据集或者你想在少量数据上运行MapReduce，Uber配置将帮助你减少MapReduce通常花费的额外时间mapper和reducers阶段。

我可以为所有MapReduce作业配置 Uber 吗？

截至目前，仅支持
仅包含地图的作业和包含一个reducer的
作业。

Hi I am a big data newbie. I searched all over the internet to find what exactly uber mode is. The more I searched the more I got confused. Can anybody please help me by answering my questions?

• What does uber mode do?
• Does it works differently in mapred 1.x and 2.x?
• And where can I find the setting for it?

解决方案

What is UBER mode in Hadoop2?

Normally mappers and reducers will run by ResourceManager (RM), RM will create separate container for mapper and reducer. Uber configuration, will allow to run mapper and reducers in the same process as the ApplicationMaster (AM).

Uber jobs :

Uber jobs are jobs that are executed within the MapReduce ApplicationMaster. Rather then communicate with RM to create the mapper and reducer containers. The AM runs the map and reduce tasks within its own process and avoided the overhead of launching and communicate with remote containers.

Why

If you have a small dataset or you want to run MapReduce on small amount of data, Uber configuration will help you out, by reducing additional time that MapReduce normally spends in mapper and reducers phase.

Can I configure an Uber for all MapReduce job?

As of now, map-only jobs and jobs with one reducer are supported.

这篇关于“超级模式”的目的是什么？在hadoop？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！