合并pySpark RDD中的列表列表 [英] Merge list of lists in pySpark RDD

问题描述

我有一些元组列表，我想合并成一个列表.我已经能够使用lambda和列表理解来处理数据，而我接近能够使用reduceByKey的地方，但是不确定如何合并列表.所以格式...

I have lists of tuples that I want to combine into one list. I've been able to process the data using lambdas and list comprehension to where I'm close to being able to use reduceByKey but not sure how to merge the lists. So the format...

[[(0, 14), (0, 24)], [(1, 19), (1, 50)], ...]

我希望这样....

[(0, 14), (0, 24), (1, 19), (1, 50), ...]

使我到达需要的地方的代码...

Code that got me to where I need to be...

test = test.map(lambda x: (x[1], [e * local[x[1]] for e in x[0]]))
test = test.map(lambda x: [(x[0], y) for y in x[1]])

但是不确定从那里开始如何合并列表

But not sure from there what to do to merge the lists

推荐答案

您可以做到，

test = test.flatMap(identity)

或

test = test.flatMap(lambda list: list)

这篇关于合并pySpark RDD中的列表列表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！