在 pyspark 中对列表中的不同数据框列求和的正确方法是什么? [英] Whats is the correct way to sum different dataframe columns in a list in pyspark?

问题描述

我想对 spark 数据框中的不同列求和.

I want to sum different columns in a spark dataframe.

代码

from pyspark.sql import functions as F
cols = ["A.p1","B.p1"]
df = spark.createDataFrame([[1,2],[4,89],[12,60]],schema=cols)

# 1. Works
df = df.withColumn('sum1', sum([df[col] for col in ["`A.p1`","`B.p1`"]]))

#2. Doesnt work
df = df.withColumn('sum1', F.sum([df[col] for col in ["`A.p1`","`B.p1`"]]))

#3. Doesnt work
df = df.withColumn('sum1', sum(df.select(["`A.p1`","`B.p1`"])))

为什么不采用#2 方法.&#3.不工作?我使用的是 Spark 2.2

Why isn't approach #2. & #3. not working? I am on Spark 2.2

推荐答案

因为，

# 1. Works
df = df.withColumn('sum1', sum([df[col] for col in ["`A.p1`","`B.p1`"]]))

这里你使用的是 python 内置的 sum 函数，它以 iterable 作为输入，所以它可以工作.https://docs.python.org/2/library/functions.html#sum

Here you are using python in-built sum function which takes iterable as input,so it works. https://docs.python.org/2/library/functions.html#sum

#2. Doesnt work
df = df.withColumn('sum1', F.sum([df[col] for col in ["`A.p1`","`B.p1`"]]))

这里您使用的是 pyspark sum 函数，该函数将列作为输入，但您正试图在行级别获取它.http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.sum

Here you are using pyspark sum function which takes column as input but you are trying to get it at row level. http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.sum

#3. Doesnt work
df = df.withColumn('sum1', sum(df.select(["`A.p1`","`B.p1`"])))

此处，df.select() 返回一个数据帧并尝试对数据帧求和.在这种情况下，我认为，您必须逐行迭代并对其应用 sum.

Here, df.select() returns a dataframe and trying to sum over a dataframe. In this case, I think, you got to iterate rowwise and apply sum over it.

这篇关于在 pyspark 中对列表中的不同数据框列求和的正确方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！