如何展平 Spark 数据帧中的结构? [英] How to flatten a struct in a Spark dataframe?

问题描述

我有一个具有以下结构的数据框:

I have a dataframe with the following structure:

 |-- data: struct (nullable = true)
 |    |-- id: long (nullable = true)
 |    |-- keyNote: struct (nullable = true)
 |    |    |-- key: string (nullable = true)
 |    |    |-- note: string (nullable = true)
 |    |-- details: map (nullable = true)
 |    |    |-- key: string
 |    |    |-- value: string (valueContainsNull = true)

如何扁平化结构并创建新的数据框:

How it is possible to flatten the structure and create a new dataframe:

     |-- id: long (nullable = true)
     |-- keyNote: struct (nullable = true)
     |    |-- key: string (nullable = true)
     |    |-- note: string (nullable = true)
     |-- details: map (nullable = true)
     |    |-- key: string
     |    |-- value: string (valueContainsNull = true)

是否有类似爆炸的东西，但对于结构而言?

Is there something like explode, but for structs?

推荐答案

这应该适用于 Spark 1.6 或更高版本:

This should work in Spark 1.6 or later:

df.select(df.col("data.*"))

或

df.select(df.col("data.id"), df.col("data.keyNote"), df.col("data.details"))

这篇关于如何展平 Spark 数据帧中的结构?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！