在Hive中，哪个查询更好，为什么? [英] In Hive, which query is better and why?

问题描述

假设有两个查询:

1. select count(distinct a) from x;

select count(*) from (select distinct a from x) y;

我知道它们返回相同的结果，但是从Hive的角度来看(使用MapReduce).谁能解释哪个是更好的选择，为什么?

I know they return the same results, but from the perspective of Hive (using MapReduce). Can anyone please explain which one is the better choice and why?

感谢您的帮助.

推荐答案

在1.2.0之前的Hive版本中，第一个查询使用一个Map和一个Reduce阶段执行. Map将每个值发送到单个化简器，化简器完成所有工作.

In Hive versions prior 1.2.0 the first query executes using one Map and one Reduce stages. Map sends each value to the single reducer, and reducer does all the job.

在这种情况下，单个reducer处理过多的数据.

Single reducer processing too much data in this case.

在第二次查询执行期间，映射器输出分布在许多化简器之间，每个化简器生成其不同的列表，最后的map-reduce作业确实汇总了每个列表的大小.

During second query execution, mappers output distributed between many reducers, each reducer generates it's distinct list and final map-reduce job does summarize the size of each list.

自Hive 1.2.0起，Hive 1.2.0+提供了自动重写优化hive.optimize.distinct.rewrite=true/false，请参见

Since Hive 1.2.0 Hive 1.2.0+ provides auto-rewrite optimization hive.optimize.distinct.rewrite=true/false, see HIVE-10568

另请参见 Hortonworks社区

这篇关于在Hive中，哪个查询更好，为什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！