比较两个数组并获得 PySpark 中的差异 [英] Comparing two arrays and getting the difference in PySpark
问题描述
我在一个数据框中有两个数组字段.
I have two array fields in a data frame.
我需要比较这两个数组并在同一数据框中将差异作为数组(新列)获取.
I have a requirement to compare these two arrays and get the difference as an array(new column) in the same data frame.
预期输出为:
B 列是 A 列的子集.此外,两个数组中的单词顺序相同.
Column B is a subset of column A. Also the words is going to be in the same order in both arrays.
有人可以帮我找到解决方案吗?
Can any one please help me to get a solution for this?
推荐答案
您可以使用用户定义的函数.我的示例数据帧与您的略有不同,但代码应该可以正常工作:
You can use a user-defined function. My example dataframe differs a bit from yours, but the code should work fine:
import pandas as pd
from pyspark.sql.types import *
#example df
df=sqlContext.createDataFrame(pd.DataFrame(data=[[["hello", "world"],
["world"]],[["sample", "overflow", "text"], ["sample", "text"]]], columns=["A", "B"]))
# define udf
differencer=udf(lambda x,y: list(set(x)-set(y)), ArrayType(StringType()))
df=df.withColumn('difference', differencer('A', 'B'))
如果存在重复项,这将不起作用,因为 set 仅保留唯一项.所以你可以修改udf如下:
This does not work if there are duplicates as set retains only uniques. So you can amend the udf as follows:
differencer=udf(lambda x,y: [elt for elt in x if elt not in y] ), ArrayType(StringType()))
这篇关于比较两个数组并获得 PySpark 中的差异的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!