使用Scala将多列转换为Spark Dataframe上的一列地图 [英] Convert multiple columns into a column of map on Spark Dataframe using Scala

问题描述

我有一个数据框，其列数可变，如 Col1、Col2、Col3.我需要使用下面的代码将 Col1 和 Col2 合并为一列数据类型映射.

I have a dataframe having variable number of columns like Col1, Col2, Col3. I need combine Col1 and Col2 into one column of data type map by using the code below.

val df_converted = df.withColumn("ConvertedCols", map(lit("Col1"), col("Col1"), lit("Col2"), col("Col2")))

但是，当我不知道列的数量和名称时，如何对所有列进行操作?

But how can I do it for all columns when I don't know the number and names of the columns?

推荐答案

一种方法是通过 flatMap 将 DataFrame 的列列表扩展为 Seq(lit(c1),col(c1), lit(c2), col(c2), ...) 并应用 Spark 的 map如下图:

One approach would be to expand the column list of the DataFrame via flatMap into a Seq(lit(c1), col(c1), lit(c2), col(c2), ...) and apply Spark's map as shown below:

import org.apache.spark.sql.functions._
import spark.implicits._

val df = Seq(
  ("a", "b", "c", "d"),
  ("e", "f", "g", "h")
).toDF("c1", "c2", "c3", "c4")

val kvCols = df.columns.flatMap(c => Seq(lit(c), col(c)))

df.withColumn("ConvertedCols", map(kvCols: _*)).show(false)
// +---+---+---+---+---------------------------------------+
// |c1 |c2 |c3 |c4 |ConvertedCols                          |
// +---+---+---+---+---------------------------------------+
// |a  |b  |c  |d  |Map(c1 -> a, c2 -> b, c3 -> c, c4 -> d)|
// |e  |f  |g  |h  |Map(c1 -> e, c2 -> f, c3 -> g, c4 -> h)|
// +---+---+---+---+---------------------------------------+

这篇关于使用Scala将多列转换为Spark Dataframe上的一列地图的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！