如何在 Spark 2 中启用 Tungsten 优化? [英] How to enable Tungsten optimization in Spark 2?

问题描述

我刚刚构建了带有 hive 支持的 Spark 2，并将其部署到带有 Hortonworks 2.3.4 的集群.但是我发现这个 Spark 2.0.3 比 HDP 2.3 附带的标准 spark 1.5.3 慢

I just built Spark 2 with hive support and deploy it to a cluster with Hortonworks 2.3.4. However I find that this Spark 2.0.3 is slower than the standard spark 1.5.3 that comes with HDP 2.3

当我检查 explain 时，似乎我的 Spark 2.0.3 没有使用钨.我是否需要创建特殊版本才能启用 Tungsten?

When I check explain it seems that my Spark 2.0.3 is not using tungsten. Do I need to create special build to enable Tungsten?

Spark 1.5.3 解释

Spark 1.5.3 Explain

== Physical Plan ==
TungstenAggregate(key=[id#2], functions=[], output=[id#2])
TungstenExchange hashpartitioning(id#2)
TungstenAggregate(key=[id#2], functions=[], output=[id#2])
HiveTableScan [id#2], (MetastoreRelation default, testing, None)

火花 2.0.3

== Physical Plan ==
*HashAggregate(keys=[id#2481], functions=[])
  +- Exchange hashpartitioning(id#2481, 72)
  +- *HashAggregate(keys=[id#2481], functions=[])
  +- HiveTableScan [id#2481], MetastoreRelation default, testing

推荐答案

仍然使用 Tungsten，class 重命名:https://github.com/apache/spark/commit/8900c8d8ff1614b5ec5a2ce213832fa13462b4d4

It still uses Tungsten, class was renamed: https://github.com/apache/spark/commit/8900c8d8ff1614b5ec5a2ce213832fa13462b4d4

这篇关于如何在 Spark 2 中启用 Tungsten 优化?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！