使用pyspark从每行的数组中获取不同的计数 [英] get distinct count from an array of each rows using pyspark

问题描述

我正在使用pyspark数据帧从每行的数组中寻找不同的计数: 输入: col1 [1,1,1] [3,4,5] [1,2,1,2]

I am looking for distinct counts from an array of each rows using pyspark dataframe: input: col1 [1,1,1] [3,4,5] [1,2,1,2]

output:
1
3
2  

I used below code but it is giving me the length of an array:
output:
3
3
4

please help me how do i achieve this using python pyspark dataframe.

slen = udf(lambda s: len(s), IntegerType())
count = Df.withColumn("Count", slen(df.col1))
count.show()

Thanks in advanced !

推荐答案

对于spark2.4 +，您可以使用array_distinct，然后获取其大小，以获取数组中不同值的计数.对于大数据，使用UDF会非常缓慢且效率低下，请始终尝试使用spark内置函数.

For spark2.4+ you can use array_distinct and then just get the size of that, to get count of distinct values in your array. Using UDF will be very slow and inefficient for big data, always try to use spark in-built functions.

https://spark.apache.org/docs/latest/api/python/pyspark.sql.html#pyspark.sql.functions.array_distinct

(欢迎来到SO)

df.show()

+------------+
|        col1|
+------------+
|   [1, 1, 1]|
|   [3, 4, 5]|
|[1, 2, 1, 2]|
+------------+

df.withColumn("count", F.size(F.array_distinct("col1"))).show()

+------------+-----+
|        col1|count|
+------------+-----+
|   [1, 1, 1]|    1|
|   [3, 4, 5]|    3|
|[1, 2, 1, 2]|    2|
+------------+-----+

这篇关于使用pyspark从每行的数组中获取不同的计数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！