如何根据基于 Pyspark 中另一列的表达式的评估有条件地替换列中的值? [英] How to conditionally replace value in a column based on evaluation of expression based on another column in Pyspark?

问题描述

import numpy as np

df = spark.createDataFrame(
    [(1, 1, None),
     (1, 2, float(5)),
     (1, 3, np.nan),
     (1, 4, None),
     (0, 5, float(10)),
     (1, 6, float('nan')),
     (0, 6, float('nan'))],
    ('session', "timestamp1", "id2"))

+-------+----------+----+
|session|timestamp1| id2|
+-------+----------+----+
|      1|         1|null|
|      1|         2| 5.0|
|      1|         3| NaN|
|      1|         4|null|
|      0|         5|10.0|
|      1|         6| NaN|
|      0|         6| NaN|
+-------+----------+----+

如何在 session==0 时用值 999 替换 timestamp1 列的值?

How to replace value of timestamp1 column with value 999 when session==0?

预期输出

+-------+----------+----+
|session|timestamp1| id2|
+-------+----------+----+
|      1|         1|null|
|      1|         2| 5.0|
|      1|         3| NaN|
|      1|         4|null|
|      0|       999|10.0|
|      1|         6| NaN|
|      0|       999| NaN|
+-------+----------+----+

是否可以在 PySpark 中使用 replace() 来实现?

Is it possible to do it using replace() in PySpark?

推荐答案

你应该使用 when (with otherwise) 函数:

You should be using the when (with otherwise) function:

from pyspark.sql.functions import when

targetDf = df.withColumn("timestamp1", \
              when(df["session"] == 0, 999).otherwise(df["timestamp1"]))

这篇关于如何根据基于 Pyspark 中另一列的表达式的评估有条件地替换列中的值?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！