有没有办法在Spark中洗牌集合 [英] Is there way how to shuffle collection in Spark

问题描述

我需要使用2.2 * 10 ^ 9行对文本文件进行随机播放.有什么方法可以将其加载到spark中，然后并行地对每个分区进行混洗(对我来说，足以在分区范围内进行混洗)然后将其溢出回到文件中?

I need to shuffle text file with 2.2*10^9 lines. Is there way how I can load it in spark, then shuffle each partition in parallel(for me it is enough to shuffle within scope of partition) and then spill it back to the file?

推荐答案

要仅在分区内随机播放，您可以执行以下操作:

To shuffle only within partitions you can do something like this:

rdd.mapPartitions(new scala.util.Random().shuffle(_))

要随机播放整个RDD，请执行以下操作:

To shuffle a whole RDD:

rdd.mapPartitions(iter => {
  val rng = new scala.util.Random()
  iter.map((rng.nextInt, _))
}).partitionBy(new HashPartitioner(rdd.partitions.size)).values

这篇关于有没有办法在Spark中洗牌集合的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！