使用“IS IN"在 2 个 Spark 数据框列之间 [英] Use "IS IN" between 2 Spark dataframe columns

问题描述

我有上面的数据框:

from pyspark.sql.types import *

rdd = sc.parallelize([
        ('ALT', ['chien', 'chat'] , 'oiseau'),
        ('ALT', ['oiseau']        , 'oiseau'),
        ('TDR', ['poule','poulet'], 'poule' ),
        ('ALT', ['ours']          , 'chien' ),
        ('ALT', ['paon']          , 'tigre' ),
        ('TDR', ['tigre','lion']  , 'lion'  ),
        ('ALT', ['chat']          ,'chien'  ),
])
schema = StructType([StructField("ClientId",StringType(),True),
                     StructField("Animaux",ArrayType(StringType(),True),True),
                     StructField("Animal",StringType(),True),])
test = rdd.toDF(schema)
test.show()

+--------+---------------+------+
|ClientId|        Animaux|Animal|
+--------+---------------+------+
|     ALT|  [chien, chat]|oiseau|
|     ALT|       [oiseau]|oiseau|
|     TDR|[poule, poulet]| poule|
|     ALT|         [ours]| chien|
|     ALT|         [paon]| tigre|
|     TDR|  [tigre, lion]|  lion|
|     ALT|         [chat]| chien|
+--------+---------------+------+

我想创建一个新列，如果Animal"列中的字符串在Animaux"列中的列表中，则该列将采用值1"，否则为0".

I want to create a new column that will take the value "1" if the string in the column "Animal" is in the list in the column "Animaux" and "0" otherwise.

我试过了:

test2=test.withColumn("isinlist", F.when("Animal in Animaux", 'ok').otherwise('pas ok'))
test2=test.withColumn("isinlist", F.when(test.Animal.isin(*test.Animaux), 'ok').otherwise('pas ok'))
test.where("Animal in (Animaux)").show()
test.where("Animal in Animaux").show()
test2=test.withColumn("isinlist", F.when(test.Animal.isin(test.Animaux), 'ok').otherwise('pas ok'))

但是都没有效果...有没有人知道如何在不使用udf的情况下做到这一点......有没有直接的方法来做到这一点?

But none of it works... Does anyone know how to do it without using udf... Is there a direct way to do it?

推荐答案

你可以使用array_contains:

from pyspark.sql.functions import expr

test.withColumn("isinlist", expr("array_contains(Animaux, Animal)")).show()
# +--------+---------------+------+--------+
# |ClientId|        Animaux|Animal|isinlist|
# +--------+---------------+------+--------+
# |     ALT|  [chien, chat]|oiseau|   false|
# |     ALT|       [oiseau]|oiseau|    true|
# |     TDR|[poule, poulet]| poule|    true|
# |     ALT|         [ours]| chien|   false|
# |     ALT|         [paon]| tigre|   false|
# |     TDR|  [tigre, lion]|  lion|    true|
# |     ALT|         [chat]| chien|   false|
# +--------+---------------+------+--------+

Source 如果一列是另一列的成员，如何过滤 Spark 数据框 zero323 (Scala).

Source How to filter Spark dataframe if one column is a member of another column by zero323 (Scala).

这篇关于使用“IS IN"在 2 个 Spark 数据框列之间的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！