使用combineByKey将输出作为（key，iterable [values]） [英] Use combineByKey to get output as (key, iterable[values])

问题描述

我试图将 RDD（key，value）转换为 RDD（key，iterable [value]） ，与 groupByKey 方法返回的输出相同。
但是，作为 groupByKey 效率不高，我试图在RDD上使用 combineByKey 来代替，它不工作。以下是使用的代码：

I am trying to transform RDD(key,value) to RDD(key,iterable[value]), same as output returned by the groupByKey method. But as groupByKey is not efficient, I am trying to use combineByKey on the RDD instead, however, it is not working. Below is the code used:

val data= List("abc,2017-10-04,15.2",
          "abc,2017-10-03,19.67", 
          "abc,2017-10-02,19.8",
          "xyz,2017-10-09,46.9", 
          "xyz,2017-10-08,48.4",
          "xyz,2017-10-07,87.5", 
          "xyz,2017-10-04,83.03", 
          "xyz,2017-10-03,83.41",
          "pqr,2017-09-30,18.18", 
          "pqr,2017-09-27,18.2", 
          "pqr,2017-09-26,19.2", 
          "pqr,2017-09-25,19.47", 
          "abc,2017-07-19,96.60",
          "abc,2017-07-18,91.68", 
          "abc,2017-07-17,91.55")
val rdd = sc.parallelize(templines)
val rows = rdd.map(line => {
  val row = line.split(",")
  ((row(0), row(1)), row(2))
})

// re partition and sort based key    
val op = rows.repartitionAndSortWithinPartitions(new CustomPartitioner(4))
val temp = op.map(f => (f._1._1, (f._1._2, f._2)))

val mergeCombiners = (t1: (String, List[String]), t2: (String, List[String])) => 
    (t1._1 + t2._1, t1._2.++(t2._2))
val mergeValue = (x: (String, List[String]), y: (String, String)) => {
  val a = x._2.+:(y._2)
  (x._1, a)
}

// createCombiner, mergeValue, mergeCombiners
val x = temp.combineByKey(
  (t1: String, t2: String) => (t1, List(t2)),
  mergeValue,
  mergeCombiners)

temp.combineByKey 给出编译时错误，I我无法得到它。

temp.combineByKey is giving compile time error, I am not able to get it.

推荐答案

如果你想得到类似于 groupByKey ，那么你绝对应该使用 groupByKey 而不是其他方法。与使用 groupByKey reduceByKey ， combineByKey 等效率更高c $ c>后跟一个聚集（给出与其他 groupBy 方法可能给出的结果相同的结果）。

If you want a output similar from what groupByKey will give you, then you should absolutely use groupByKey and not some other method. The reduceByKey, combineByKey, etc. are only more efficient compared to using groupByKey followed with an aggregation (giving you the same result as one of the other groupBy methods could have given).

由于想要的结果是 RDD [key，iterable [value]] ，所以自己构建列表或让 groupByKey 这样做会导致相同数量的工作。没有必要自己重新实现 groupByKey 。 groupByKey 的问题不在于它的实现，而在于分布式架构。

As the wanted result is an RDD[key,iterable[value]], building the list yourself or letting groupByKey do it will result in the same amount of work. There is no need to reimplement groupByKey yourself. The problem with groupByKey is not its implementation but lies in the distributed architecture.

有关 groupByKey 和这些类型的优化，我建议阅读更多这里。

For more information regarding groupByKey and these types of optimizations, I would recommend reading more here.

这篇关于使用combineByKey将输出作为（key，iterable [values]）的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！