行模式作为 PySpark DataFrame 中的新列 [英] Mode of row as a new column in PySpark DataFrame

问题描述

是否可以根据前一列的最大值添加新列，其中前一列是字符串文字.考虑以下数据框:

Is it possible to add a new column based on the maximum of previous columns where the previous columns are string literals. Consider following dataframe:

df = spark.createDataFrame(
    [
        ('1',25000,"black","black","white"),
        ('2',16000,"red","black","white"),
    ],
    ['ID','cash','colour_body','colour_head','colour_foot']
)

那么目标框架应该是这样的:

Then the target frame should look like this:

df = spark.createDataFrame(
    [
        ('1',25000,"black","black","white", "black" ),
        ('2',16000,"red","black","white", "white" ),
    ],
    ['ID','cash','colour_body','colour_head','colour_foot', 'max_v']
)

如果没有可检测的最大值，则应使用最后一个有效颜色.

If there is no maximum detectable, then the last valid colour should be used.

是否有某种计数器可用或 udf 的可能性?

Is there some kind of counter possibility available or udf?

推荐答案

定义一个围绕 statistics.mode 的 UDF 来计算具有所需语义的行模式:

Define a UDF around statistics.mode to compute the row-wise mode with the required semantics:

import statistics

from pyspark.sql.functions import udf, col
from pyspark.sql.types import StringType

def mode(*x):
    try:
        return statistics.mode(x)
    except statistics.StatisticsError:
        return x[-1]

mode = udf(mode, StringType())

df.withColumn("max_v", mode(*[col(c) for c in df.columns if 'colour' in c])).show()

+---+-----+-----------+-----------+-----------+-----+
| ID| cash|colour_body|colour_head|colour_foot|max_v|
+---+-----+-----------+-----------+-----------+-----+
|  1|25000|      black|      black|      white|black|
|  2|16000|        red|      black|      white|white|
+---+-----+-----------+-----------+-----------+-----+

这篇关于行模式作为 PySpark DataFrame 中的新列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！