reduceByKey：它是如何工作的内部？ [英] reduceByKey: How does it work internally?

问题描述

我是新来激发和Scala。我感到困惑的火花方式reuceByKey功能的作品。假设我们哈瓦以下code：

I am new to spark and scala. I was confused about the way reuceByKey function works in spark. Suppose we hava the following code:

    val lines = sc.textFile("data.txt")
    val pairs = lines.map(s => (s, 1))
    val counts = pairs.reduceByKey((a, b) => a + b)

地图功能是明确的：s是关键的，它指出从data.txt中的线和1的值

the map function is clear: s is the key and it points to the line from data.txt and 1 is the value.

不过，我没有得到reduceByKey内部是如何工作？请问A点的关键？可替代地，确实的a点的？那么，什么是重present A + B？他们如何填补？

However, I didn't get how the reduceByKey works internally? Does "a" points to the key ? alternatively, does "a" point to "s" ? Then what does represent a + b ? how are they filled ?

推荐答案

让我们把它分解到离散方法和类型。这通常暴露出新的开发者的复杂性：

Let's break it down to discrete methods and types. That usually exposes the intricacies for new devs:

pairs.reduceByKey((a, b) => a + b)

变为

pairs.reduceByKey((a: Int, b: Int) => a + b)

和重命名变量使得它多一点明确的

and renaming the variables makes it a little more explicit

pairs.reduceByKey((accumulatedValue: Int, currentValue: Int) => accumulatedValue + currentValue)

所以，我们现在可以看到，我们正在采取简单的累计值给定键并与该键的下一个值的总结它。现在，让我们进一步打破，所以我们能够理解的关键部件。所以，让我们直观的方法更是这样的：

So, we can now see that we are simply taking an accumulated value for the given key and summing it with the next value of that key. NOW, let's break it further so we can understand the key part. So, let's visualize the method more like this:

pairs.reduce((accumulatedValue: List[(String, Int)], currentValue: (String, Int)) => {
  //Turn the accumulated value into a true key->value mapping
  val accumAsMap = accumulatedValue.toMap   
  //Try to get the key's current value if we've already encountered it
  accumAsMap.get(currentValue._1) match { 
    //If we have encountered it, then add the new value to the existing value and overwrite the old
    case Some(value : Int) => (accumAsMap + (curr._1 -> (value + curr._2))).toList
    //If we have NOT encountered it, then simply add it to the list
    case None => curr :: accumulatedValue 
  }
})

所以，你可以看到，减少的 ByKey 需要找到钥匙，跟踪它的样板，这样你就不必担心管理的那部分。

So, you can see that the reduceByKey takes the boilerplate of finding the key and tracking it so that you don't have to worry about managing that part.

更深，更真实，如果你想

之所以这么说，这是为有一些优化，在这里所做的一切发生在一个简化版本。此操作是相联的，所以火花引擎将首先在本地执行这些削减的驱动程序（通常称为地图边减少），然后再次。这样可以节省网络流量;而不是发送的所有数据和执行操作，它可以减少它小，因为它可以再发送通过线路到减少。

All that being said, that is a simplified version of what happens as there are some optimizations that are done here. This operation is associative, so the spark engine will perform these reductions locally first (often termed map-side reduce) and then once again at the driver. This saves network traffic; instead of sending all the data and performing the operation, it can reduce it as small as it can and then send that reduction over the wire.

这篇关于reduceByKey：它是如何工作的内部？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！