在pyspark中的列表中汇总不同数据框列的正确方法是什么? [英] Whats is the correct way to sum different dataframe columns in a list in pyspark?
问题描述
我想对spark数据框中的不同列求和.
I want to sum different columns in a spark dataframe.
代码
from pyspark.sql import functions as F
cols = ["A.p1","B.p1"]
df = spark.createDataFrame([[1,2],[4,89],[12,60]],schema=cols)
# 1. Works
df = df.withColumn('sum1', sum([df[col] for col in ["`A.p1`","`B.p1`"]]))
#2. Doesnt work
df = df.withColumn('sum1', F.sum([df[col] for col in ["`A.p1`","`B.p1`"]]))
#3. Doesnt work
df = df.withColumn('sum1', sum(df.select(["`A.p1`","`B.p1`"])))
为什么不采用第二种方法. & #3.不工作吗? 我正在使用Spark 2.2
Why isn't approach #2. & #3. not working? I am on Spark 2.2
推荐答案
因为
# 1. Works
df = df.withColumn('sum1', sum([df[col] for col in ["`A.p1`","`B.p1`"]]))
在这里,您使用的是python内置求和函数,该函数将Iterable作为输入,因此可以正常工作. https://docs.python.org/2/library/functions.html#sum
Here you are using python in-built sum function which takes iterable as input,so it works. https://docs.python.org/2/library/functions.html#sum
#2. Doesnt work
df = df.withColumn('sum1', F.sum([df[col] for col in ["`A.p1`","`B.p1`"]]))
这里您正在使用pyspark sum函数,该函数将列作为输入,但是您试图在行级获取它. http://spark.apache .org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.sum
Here you are using pyspark sum function which takes column as input but you are trying to get it at row level. http://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.sum
#3. Doesnt work
df = df.withColumn('sum1', sum(df.select(["`A.p1`","`B.p1`"])))
在这里,df.select()返回一个数据框并尝试对一个数据框求和.我认为,在这种情况下,您必须逐行迭代并将总和应用于其上.
Here, df.select() returns a dataframe and trying to sum over a dataframe. In this case, I think, you got to iterate rowwise and apply sum over it.
这篇关于在pyspark中的列表中汇总不同数据框列的正确方法是什么?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!