Pyspark:将具有嵌套结构的数组转换为字符串 [英] Pyspark: cast array with nested struct to string

问题描述

我有pyspark数据框，其中的列名为过滤器: "array>"

I have pyspark dataframe with a column named Filters: "array>"

我想将数据帧保存在csv文件中，为此我需要将数组转换为字符串类型.

I want to save my dataframe in csv file, for that i need to cast the array to string type.

我尝试将其强制转换为DF.Filters.tostring()和DF.Filters.cast(StringType())，但是两种解决方案均会在过滤器"列中为每一行生成错误消息:

I tried to cast it: DF.Filters.tostring() and DF.Filters.cast(StringType()), but both solutions generate error message for each row in the columns Filters:

org.apache.spark.sql.catalyst.expressions.UnsafeArrayData@56234c19

代码如下

from pyspark.sql.types import StringType

DF.printSchema()

|-- ClientNum: string (nullable = true)
|-- Filters: array (nullable = true)
    |-- element: struct (containsNull = true)
          |-- Op: string (nullable = true)
          |-- Type: string (nullable = true)
          |-- Val: string (nullable = true)

DF_cast = DF.select ('ClientNum',DF.Filters.cast(StringType())) 

DF_cast.printSchema()

|-- ClientNum: string (nullable = true)
|-- Filters: string (nullable = true)

DF_cast.show()

| ClientNum | Filters 
|  32103    | org.apache.spark.sql.catalyst.expressions.UnsafeArrayData@d9e517ce
|  218056   | org.apache.spark.sql.catalyst.expressions.UnsafeArrayData@3c744494

示例JSON数据:

{"ClientNum":"abc123","Filters":[{"Op":"foo","Type":"bar","Val":"baz"}]}

谢谢!!

推荐答案

我创建了一个示例JSON数据集来匹配该模式:

I created a sample JSON dataset to match that schema:

{"ClientNum":"abc123","Filters":[{"Op":"foo","Type":"bar","Val":"baz"}]}

select(s.col("ClientNum"),s.col("Filters").cast(StringType)).show(false)

+---------+------------------------------------------------------------------+
|ClientNum|Filters                                                           |
+---------+------------------------------------------------------------------+
|abc123   |org.apache.spark.sql.catalyst.expressions.UnsafeArrayData@60fca57e|
+---------+------------------------------------------------------------------+

使用explode()函数可以最佳化您的问题，该函数可以展平数组，然后使用星号展开符号:

Your problem is best solved using the explode() function which flattens an array, then the star expand notation:

s.selectExpr("explode(Filters) AS structCol").selectExpr("structCol.*").show()
+---+----+---+
| Op|Type|Val|
+---+----+---+
|foo| bar|baz|
+---+----+---+

将其设置为由逗号分隔的单列字符串:

To make it a single column string separated by commas:

s.selectExpr("explode(Filters) AS structCol").select(F.expr("concat_ws(',', structCol.*)").alias("single_col")).show()
+-----------+
| single_col|
+-----------+
|foo,bar,baz|
+-----------+

分解数组参考:在Spark中平整行

结构"类型的星标扩展参考:如何在Spark数据框中展平结构?

Star expand reference for "struct" type: How to flatten a struct in a spark dataframe?

这篇关于Pyspark:将具有嵌套结构的数组转换为字符串的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！