等价于IF的火花等于ELSE [英] Spark Equivalent of IF Then ELSE
本文介绍了等价于IF的火花等于ELSE的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我在这里早些时候已经看到了这个问题,并从中吸取了教训.但是,我不确定为什么感觉应该可以工作时出现错误.
I have seen this question earlier here and I have took lessons from that. However I am not sure why I am getting an error when I feel it should work.
我想通过一些规则在现有Spark DataFrame
中创建一个新列.这是我写的. iris_spark是具有类别变量iris_spark的数据框,该变量具有三个不同的类别.
I want to create a new column in existing Spark DataFrame
by some rules. Here is what I wrote. iris_spark is the data frame with a categorical variable iris_spark with three distinct categories.
from pyspark.sql import functions as F
iris_spark_df = iris_spark.withColumn(
"Class",
F.when(iris_spark.iris_class == 'Iris-setosa', 0, F.when(iris_spark.iris_class == 'Iris-versicolor',1)).otherwise(2))
引发以下错误.
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-157-21818c7dc060> in <module>()
----> 1 iris_spark_df=iris_spark.withColumn("Class",F.when(iris_spark.iris_class=='Iris-setosa',0,F.when(iris_spark.iris_class=='Iris-versicolor',1)))
TypeError: when() takes exactly 2 arguments (3 given)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-157-21818c7dc060> in <module>()
----> 1 iris_spark_df=iris_spark.withColumn("Class",F.when(iris_spark.iris_class=='Iris-setosa',0,F.when(iris_spark.iris_class=='Iris-versicolor',1)))
TypeError: when() takes exactly 2 arguments (3 given)
知道为什么吗?
推荐答案
正确的结构是:
(when(col("iris_class") == 'Iris-setosa', 0)
.when(col("iris_class") == 'Iris-versicolor', 1)
.otherwise(2))
等效于
CASE
WHEN (iris_class = 'Iris-setosa') THEN 0
WHEN (iris_class = 'Iris-versicolor') THEN 1
ELSE 2
END
或:
(when(col("iris_class") == 'Iris-setosa', 0)
.otherwise(when(col("iris_class") == 'Iris-versicolor', 1)
.otherwise(2)))
等效于:
CASE WHEN (iris_class = 'Iris-setosa') THEN 0
ELSE CASE WHEN (iris_class = 'Iris-versicolor') THEN 1
ELSE 2
END
END
具有通用语法:
when(condition, value).when(...)
或
when(condition, value).otherwise(...)
您可能将Hive IF
有条件地混为一谈:
You probably mixed up things with Hive IF
conditional:
IF(condition, if-true, if-false)
只能在具有Hive支持的原始SQL中使用.
which can be used only in raw SQL with Hive support.
这篇关于等价于IF的火花等于ELSE的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文