如何有效地在星火每一列找到不同的值 [英] How to efficiently find distinct values from each column in Spark

问题描述

为了从阵列中的每一列找到不同的值我试过

RDD[Array[String]].map(_.map(Set(_))).reduce { 
(a, b) => (a.zip(b)).map { case (x, y) => x ++ y}}

成功地执行。不过，我想知道是否有这样做的比我的样本code以上的更有效的方式。谢谢你。

which executes successfully. However, I'd like to know if there was a more efficient way of doing this than my sample code above. Thank you.

推荐答案

总结节省了一步，可能会或可能不会是更有效的。

Aggregate saves a step, might or might not be more efficient

val z = Array.fill(5)(Set[String]()) // or whatever the length is
val d= lists.aggregate(z)({(a, b) => (a.zip(b)).map { case (x, y) => x + y}}, 
                          {(a, b) => (a.zip(b)).map { case (x, y) => x ++ y}})

您也可以尝试使用可变套和修改，而不是产生在每一步一个新的（这是明确允许星火）：

You could also try using mutable sets and modifying rather than producing a new one at each step (which is explicitly allowed by Spark):

val z = Array.fill(5)(scala.collection.mutable.Set[String]())
val d= lists.aggregate(z)({(a, b) => (a.zip(b)).foreach { case (x, y) => x+= y };a},
                          {(a, b) => (a.zip(b)).foreach { case (x, y) => x ++= y};a})

这篇关于如何有效地在星火每一列找到不同的值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！