重命名spark数据框structType字段 [英] rename spark dataframe structType fields

问题描述

给出一个动态的structType.在这里structType名称是未知的.它是动态的，因此它的名称正在更改.

Given a dynamic structType . here structType name is not known . It is dynamic and hence its name is changing.

名称是变量.因此，请勿预先假设"MAIN_COL"在模式中.

The name is variable . So don't pre assume "MAIN_COL" in the schema.

root
 |-- MAIN_COL: struct (nullable = true)
 |    |-- a: string (nullable = true)
 |    |-- b: string (nullable = true)
 |    |-- c: string (nullable = true)
 |    |-- d: string (nullable = true)
 |    |-- f: long (nullable = true)
 |    |-- g: long (nullable = true)
 |    |-- h: long (nullable = true)
 |    |-- j: long (nullable = true)

我们如何编写动态代码来重命名以其名称作为前缀的structType的字段.

how can we write a dynamic code to rename the fields of a structType with its name as its prefix.

root
 |-- MAIN_COL: struct (nullable = true)
 |    |-- MAIN_COL_a: string (nullable = true)
 |    |-- MAIN_COL_b: string (nullable = true)
 |    |-- MAIN_COL_c: string (nullable = true)
 |    |-- MAIN_COL_d: string (nullable = true)
 |    |-- MAIN_COL_f: long (nullable = true)
 |    |-- MAIN_COL_g: long (nullable = true)
 |    |-- MAIN_COL_h: long (nullable = true)
 |    |-- MAIN_COL_j: long (nullable = true)

推荐答案

您可以使用DSL更新嵌套列的架构.

You can use DSL to update the schema of nested columns.

import org.apache.spark.sql.types._

val schema: StructType = df.schema.fields.head.dataType.asInstanceOf[StructType]

val updatedSchema = StructType.apply(
       schema.fields.map(sf => StructField.apply("MAIN_COL_" + sf.name, sf.dataType))
)

val resultDF = df.withColumn("MAIN_COL", $"MAIN_COL".cast(updatedSchema))

更新的架构:

root
 |-- MAIN_COL: struct (nullable = false)
 |    |-- MAIN_COL_a: string (nullable = true)
 |    |-- MAIN_COL_b: string (nullable = true)
 |    |-- MAIN_COL_c: string (nullable = true)

这篇关于重命名spark数据框structType字段的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！