Spark 是否有关于 RDD 的最佳分区数量及其元素数量的经验法则? [英] Spark Is there any rule of thumb about the optimal number of partition of a RDD and its number of elements?

问题描述

RDD 包含的元素数量与其理想的分区数量之间有什么关系?

Is there any relationship between the number of elements an RDD contained and its ideal number of partitions ?

我有一个包含数千个分区的 RDD(因为我从一个由多个小文件组成的源文件加载它，这是一个我无法修复的约束，所以我必须处理它).我想对其重新分区(或使用 coalesce 方法).但我事先不知道 RDD 将包含的事件的确切数量.
所以我想以自动化的方式来做.看起来像:

I have a RDD that has thousand of partitions (because I load it from a source file composed by multiple small files, that's a constraint I can't fix so I have to deal with it). I would like to repartition it (or use the coalescemethod). But I don't know in advance the exact number of events the RDD will contain.
So I would like to do it in an automated way. Something that will look like:

val numberOfElements = rdd.count()
val magicNumber = 100000
rdd.coalesce( numberOfElements / magicNumber)

关于 RDD 的最佳分区数量及其元素数量是否有任何经验法则?

Is there any rule of thumb about the optimal number of partition of a RDD and its number of elements ?

谢谢.

推荐答案

没有，因为它高度依赖于应用程序、资源和数据.有一些硬性限制(比如各种 2GB 限制)，但其余的你有以任务为基础进行调整.需要考虑的一些因素:

There isn't, because it is highly dependent on application, resources and data. There are some hard limitations (like various 2GB limits) but the rest you have to tune on task to task basis. Some factors to consider:

• 单行/元素的大小
• 典型操作的成本.如果分区较小且操作成本低，则调度成本可能远高于数据处理成本.
• 执行分区(例如排序)操作时处理分区的成本.

如果这里的核心问题是一些初始文件，那么使用 CombineFileInputFormat 的一些变体可能比重新分区/合并更好.例如:

If the core problem here is a number of the initial files then using some variant of CombineFileInputFormat could be a better idea than repartitioning / coalescing. For example:

sc.hadoopFile(
  path,
  classOf[CombineTextInputFormat],
  classOf[LongWritable], classOf[Text]
).map(_._2.toString)

另请参阅如何计算合并的最佳分区数?

这篇关于Spark 是否有关于 RDD 的最佳分区数量及其元素数量的经验法则?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！