Sqoop 导入.可以执行多少个最大映射器 [英] Sqoop import. how many max mapper could be executed

问题描述

在 Sqoop 导入中可以执行多少个最大数量的映射器.此外，在使用 sqoop 导入时，是否有任何减速器正在运行的情况.

解决方案

1.Sqoop 导入中最多可以执行多少个 mapper?

<块引用><块引用>

增加映射器的数量会导致更多的并发数据传输任务，'可以'导致更快的工作完成.

它并不总是能更快地完成工作.同时增加映射器的数量，有一个点你会完全饱和你的数据库.增加映射器的数量超过这一点不会导致更快的工作完成；事实上，它将具有相反的效果，因为您的数据库服务器花费更多时间处理上下文切换而不是提供数据.

映射器的最佳数量取决于许多变量:

1.数据库类型.

2. 用于您的数据库服务器的硬件.

3. 影响您的数据库需要处理的其他请求.<块引用>

   从少量的映射器开始，让您找到最佳的您的环境和用例的并行度.

2.另外，在使用 sqoop 导入时，是否有任何 reducer 是运行.

<块引用>

聚合需要减速器.sqoop的减速器数量为0，因为它只是一个运行 MAP 作业的作业，该作业将数据转储到高清文件系统.我们没有聚合任何东西.

How many max number of mapper could be executed in Sqoop import. Also, while importing using sqoop is there any case where reducer is running.

解决方案

1.How many max number of mapper could be executed in Sqoop import?

Increasing the number of mappers will lead to a higher number of concurrent data transfer tasks, 'which can' result in faster job completion.

It won’t always lead to faster job completion. While increasing the number of mappers, there is a point at which you will fully saturate your database. Increasing the number of mappers beyond this point won’t lead to faster job completion; in fact, it will have the opposite effect as your database server spends more time doing context switching rather than serving data.

The optimal number of mappers depends on many variables:

1.Database type.

2.Hardware that is used for your database server.

3. Impact to other requests that your database needs to serve.

   Start with small number of mappers for your to find the optimal degree of parallelism for your environment and use case.

2.Also, while importing using sqoop is there any case where reducer is running.

Reducers are needed for aggregation.Number of reducers for sqoop is 0, since it is merely a job running a MAP only job that dumps data into HDFS. We are not aggregating anything.

这篇关于Sqoop 导入.可以执行多少个最大映射器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！