PySpark:DataFrame-将结构转换为数组 [英] PySpark: DataFrame - Convert Struct to Array

问题描述

我具有以下结构的数据框:

I have a dataframe in the following structure:

root
 |-- index: long (nullable = true)
 |-- text: string (nullable = true)
 |-- topicDistribution: struct (nullable = true)
 |    |-- type: long (nullable = true)
 |    |-- values: array (nullable = true)
 |    |    |-- element: double (containsNull = true)
 |-- wiki_index: string (nullable = true)

我需要将其更改为:

root
 |-- index: long (nullable = true)
 |-- text: string (nullable = true)
 |-- topicDistribution: array (nullable = true)
 |    |--  element: double (containsNull = true)
 |-- wiki_index: string (nullable = true)

请问我该怎么做?

非常感谢.

推荐答案

我认为您正在寻找

df.withColumn("topicDistribution", col("topicDistribution").getField("values"))

这篇关于PySpark:DataFrame-将结构转换为数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！