Pyspark:如何根据另一列中的匹配值从数组中的第一个匹配项中选择直到最后一个值的值 [英] Pyspark : How to pick the values till last from the first occurrence in an array based on the matching values in another column

问题描述

我有一个数据框，我需要搜索一个列中存在的值，即另一列中的StringType，即ArrayType，但是我想从第二列中选择值，直到从第一列的第一个出现到数组中的最后一个值柱子.

I have an dataframe where I need to search a value present in one column i.e., StringType in another column i.e., ArrayType but I want to pick the values from the second column till last value in array from the first occurences of the first column.

下面举例说明:

输入DF如下:

Employee_Name|Employee_ID|Mapped_Project_ID
Name1|E101|[E101, E102, E103]
Name2|E102|[E101, E102, E103]
Name3|E103|[E101, E102, E103, E104, E105] 

输出DF应该如下所示:

Output DF Should look like as below:

Employee_Name|Employee_ID|Mapped_Project_ID
Name1|E101|[E101, E102, E103]
Name2|E102|[E102, E103]
Name3|E103|[E103, E104, E105] 

推荐答案

从Spark 2.4开始，您可以使用array_position和slice函数:

As of Spark 2.4 you can use array_position and slice functions:

import pyspark.sql.functions as f    
from pyspark.sql.functions import array_position
from pyspark.sql.functions import slice

df = spark.createDataFrame([(["c", "b", "a","e","f"],'a')], ['arraydata','item'])

df.select(df.arraydata, f.expr("slice(arraydata,array_position(arraydata, item),size(arraydata))").alias("res")).show()

+---------------+---------+
|      arraydata|      res|
+---------------+---------+
|[c, b, a, e, f]|[a, e, f]|
+---------------+---------+

请仅将其翻译为您的df姓氏.希望这会有所帮助.

Please just translate this into your df colnames. Hope this helps.

这篇关于Pyspark:如何根据另一列中的匹配值从数组中的第一个匹配项中选择直到最后一个值的值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！