其作用火花所使用的按键两结合RDDS [英] Which function in spark is used to combine two RDDs by keys

问题描述

让我们说，我有以下两个RDDS，具有以下密钥对的值。

Let us say I have the following two RDDs, with the following key-pair values.

rdd1 = [ (key1, [value1, value2]), (key2, [value3, value4]) ]

和

rdd2 = [ (key1, [value5, value6]), (key2, [value7]) ]

现在，我想通过键值加入他们，所以比如我想返回以下

Now, I want to join them by key values, so for example I want to return the following

ret = [ (key1, [value1, value2, value5, value6]), (key2, [value3, value4, value7]) ] 

我如何我能做到这一点，火花使用python或Scala呢？一种方法是使用加入，但加入将创建的元组内的元组。但我想只有每个键值对一个元组。

How I can I do this, in spark using python or scala? One way is to use join, but join would create a tuple inside the tuple. But I want to only have one tuple per key value pair.

推荐答案

我想联盟两个RDDS和一个reduceByKey合并值。

I would union the two RDDs and to a reduceByKey to merge the values.

(rdd1 union rdd2).reduceByKey(_ ++ _)

这篇关于其作用火花所使用的按键两结合RDDS的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！