如何对RDD中的列的值求和 [英] How to Sum values of Column Within RDD

问题描述

我有一个带有以下行的RDD:

I have an RDD with the following rows:

[(id,value)]

如何对RDD中所有行的值求和?

How would you sum the values of all rows in the RDD?

推荐答案

只需使用

Simply use sum, you just need to get the data into a list.

例如

sc.parallelize([('id', [1, 2, 3]), ('id2', [3, 4, 5])]) \ 
    .flatMap(lambda tup: tup[1]) \ # [1, 2, 3, 3, 4, 5]
    .sum()

输出18

类似地，只需使用 values()本身就可以将第二列作为RDD获得.

Similarly, just use values() to get that second column as an RDD on it's own.

sc.parallelize([('id', 6), ('id2', 12)]) \ 
    .values() \ # [6, 12]
    .sum()

这篇关于如何对RDD中的列的值求和的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！