重命名Scala Spark Dataframe中的嵌套元素 [英] Renaming nested elements in Scala Spark Dataframe

问题描述

我有一个带有嵌套结构的 Spark Scala 数据框:

I have a Spark Scala dataframe with a nested structure:

 |-- _History: struct (nullable = true)
 |    |-- Article: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- Id: string (nullable = true)
 |    |    |    |-- Timestamp: long (nullable = true)
 |    |-- Channel: struct (nullable = true)
 |    |    |-- <font><font>Cultura pop</font></font>: array (nullable = true)
 |    |    |    |-- element: long (containsNull = true)
 |    |    |-- <font><font>Deportes</font></font>: array (nullable = true)
 |    |    |    |-- element: long (containsNull = true)

我正在尝试重命名嵌套元素(例如 Deportes 到 Deportes.是有没有办法使用 UDF 或类似的东西来做到这一点?

I'm trying to rename the nested elements (e.g. <font><font>Deportes</font></font> to Deportes. Is there a way to do this using a UDF or something similar?

我尝试了以下方法，但不起作用:

I've tried the following, which doesn't work:

var filterDF2 = filterDF
  .withColumnRenamed("_History.Channel.<font><font>Deportes</font></font>", "_History.Channel.Deportes")

推荐答案

最简单的方法是使用类型转换和正确命名的模式字符串(或等效的 StructField 定义):

The simplest approach is to use type casting with properly named schema string (or equivalent StructField definition):

val schema = """struct<
  Article: array<struct<Id:string,Timestamp:bigint>>,
  Channel: struct<Cultura: bigint, Deportes: array<bigint>>>"""
df.withColumn("_History", $"_History".cast(schema))

您也可以使用案例类对此进行建模:

You could also model this with case classes:

import org.apache.spark.sql.Row

case class ChannelRecord(Cultura: Option[Long], Deoprtes: Option[Seq[Long]])

val rename = udf((row: Row) => 
  ChannelRecord(Option(row.getLong(0)), Option(row.getSeq[Long](1))))

df.withColumn("_History",
  struct($"_History.Article", rename($"_History.channel").alias("channel")))

这篇关于重命名Scala Spark Dataframe中的嵌套元素的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！