ValueError:无法将列转换为布尔值 [英] ValueError: Cannot convert column into bool

问题描述

我正在尝试在数据框上建立一个新列，如下所示:

I'm trying build a new column on dataframe as below:

l = [(2, 1), (1,1)]
df = spark.createDataFrame(l)

def calc_dif(x,y):
    if (x>y) and (x==1):
        return x-y

dfNew = df.withColumn("calc", calc_dif(df["_1"], df["_2"]))
dfNew.show()

但是，我得到了:

Traceback (most recent call last):
  File "/tmp/zeppelin_pyspark-2807412651452069487.py", line 346, in <module>
Exception: Traceback (most recent call last):
  File "/tmp/zeppelin_pyspark-2807412651452069487.py", line 334, in <module>
  File "<stdin>", line 38, in <module>
  File "<stdin>", line 36, in calc_dif
  File "/usr/hdp/current/spark2-client/python/pyspark/sql/column.py", line 426, in __nonzero__
    raise ValueError("Cannot convert column into bool: please use '&' for 'and', '|' for 'or', "
ValueError: Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.

为什么会这样?我该如何解决?

Why It happens? How can I fix It?

推荐答案

要么使用udf:

from pyspark.sql.functions import udf

@udf("integer")
def calc_dif(x,y):
    if (x>y) and (x==1):
        return x-y

或者(推荐)的情况

from pyspark.sql.functions import when

def calc_dif(x,y):
    when(( x > y) & (x == 1), x - y)

第一个在Python对象上进行计算，第二个在Spark Columns

The first one computes on Python objects, the second one on Spark Columns

这篇关于ValueError:无法将列转换为布尔值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！