spark.sql.shuffle.partitions 的最佳值应该是多少，或者在使用 Spark SQL 时我们如何增加分区? [英] What should be the optimal value for spark.sql.shuffle.partitions or how do we increase partitions when using Spark SQL?

问题描述

我实际上使用的是 Spark SQL hiveContext.sql() 它使用 group by 查询，但我遇到了 OOM 问题.因此，考虑将 spark.sql.shuffle.partitions 的值从默认值 200 增加到 1000，但这无济于事.

我相信这个分区将共享数据随机加载，因此更多的分区可以容纳更少的数据.我是 Spark 的新手.我使用的是 Spark 1.4.0，我有大约 1TB 的未压缩数据可以使用 hiveContext.sql() 按查询分组进行处理.

解决方案

如果 shuffle 内存不足，请尝试将 spark.sql.shuffle.partitions 设置为 2001.

当分区数大于2000时，Spark使用不同的数据结构进行shuffle bookkeeping:

private[spark] object MapStatus {def apply(loc: BlockManagerId, uncompressedSizes: Array[Long]): MapStatus = {如果 (uncompressedSizes.length > 2000) {HighCompressedMapStatus(loc, uncompressedSizes)} 别的 {新的 CompressedMapStatus(loc, uncompressedSizes)}}...

我真的希望他们能让你独立配置.

顺便说一下，我找到了 Cloudera 幻灯片中的此信息.

I am using Spark SQL actually hiveContext.sql() which uses group by queries and I am running into OOM issues. So thinking of increasing value of spark.sql.shuffle.partitions from 200 default to 1000 but it is not helping.

I believe this partition will share data shuffle load so more the partitions less data to hold. I am new to Spark. I am using Spark 1.4.0 and I have around 1TB of uncompressed data to process using hiveContext.sql() group by queries.

解决方案

If you're running out of memory on the shuffle, try setting spark.sql.shuffle.partitions to 2001.

Spark uses a different data structure for shuffle book-keeping when the number of partitions is greater than 2000:

private[spark] object MapStatus {

  def apply(loc: BlockManagerId, uncompressedSizes: Array[Long]): MapStatus = {
    if (uncompressedSizes.length > 2000) {
      HighlyCompressedMapStatus(loc, uncompressedSizes)
    } else {
      new CompressedMapStatus(loc, uncompressedSizes)
    }
  }
...

I really wish they would let you configure this independently.

By the way, I found this information in a Cloudera slide deck.

这篇关于spark.sql.shuffle.partitions 的最佳值应该是多少，或者在使用 Spark SQL 时我们如何增加分区?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！