Pyspark合并数据框中的WrappedArrays [英] Pyspark Merge WrappedArrays Within a Dataframe

问题描述

当前的Pyspark数据帧具有以下结构(col2的WrappedArrays列表):

The current Pyspark dataframe has this structure (a list of WrappedArrays for col2):

+---+---------------------------------------------------------------------+  
|id |col2                                                                 |   
+---+---------------------------------------------------------------------+  
|a  |[WrappedArray(code2), WrappedArray(code1, code3)]                    |  
+---+---------------------------------------------------------------------+  
|b  |[WrappedArray(code5), WrappedArray(code6, code8)]                    |  
+---+---------------------------------------------------------------------+  

这是我想要的结构(col2的扁平列表):

This is the structure I would like to have (a flattened list for col2):

+---+---------------------------------------------------------------------+  
|id |col2                                                                 |   
+---+---------------------------------------------------------------------+  
|a  |[code2,code1, code3)]                                                |  
+---+---------------------------------------------------------------------+  
|b  |[code5,code6, code8]                                                 |  
+---+---------------------------------------------------------------------+  

但是我不确定如何进行转换.我曾尝试制作平面图，但是那似乎没有用.有什么建议吗?

but I'm not sure how to do that transformation. I had tried to do a flatmap but that didn't seem to work. Any suggestions?

推荐答案

您可以使用udf和rdd这两种方式进行此操作.这是示例:-

You can do this using 2 ways, udf and rdd. Here is example:-

df = sqlContext.createDataFrame([
    ['a',  [['code2'],['code1', 'code3']]],  
    ['b',  [['code5','code6'], ['code8']]]
], ["id", "col2"])  
df.show(truncate = False)
+---+-------------------------------------------------+
|id |col2                                             |
+---+-------------------------------------------------+
|a  |[WrappedArray(code2), WrappedArray(code1, code3)]|
|b  |[WrappedArray(code5, code6), WrappedArray(code8)]|
+---+-------------------------------------------------+

RDD:-

df.map(lambda row:(row[0], reduce(lambda x,y:x+y, row[1]))).toDF().show(truncate=False)
+---+---------------------+
|_1 |_2                   |
+---+---------------------+
|a  |[code2, code1, code3]|
|b  |[code5, code6, code8]|
+---+---------------------+

UDF:-

from pyspark.sql import functions as F
import pyspark.sql.types as T
def fudf(val):
    #emlist = []
    #for item in val:
    #    emlist += item
    #return emlist
    return reduce (lambda x, y:x+y, val)
flattenUdf = F.udf(fudf, T.ArrayType(T.StringType()))
df.select("id", flattenUdf("col2").alias("col2")).show(truncate=False)
+---+---------------------+
|id |col2                 |
+---+---------------------+
|a  |[code2, code1, code3]|
|b  |[code5, code6, code8]|
+---+---------------------+

这篇关于Pyspark合并数据框中的WrappedArrays的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！