从哈希映射创建数据帧，其中键作为列名称，值作为Spark中的行 [英] Create a dataframe from a hashmap with keys as column names and values as rows in Spark

问题描述

我有一个数据框，并且有一个列，它是这样的数据框中的映射-

I have a dataframe and I have a column which is a map in dataframe like this -

scala> df.printSchema

root
 |-- A1: map (nullable = true)
 |    |-- key: string
 |    |-- value: string (valueContainsNull = true)

我需要从数据框中选择所有键作为列名，将值选择为行.

I need to select all the keys from dataframe as column name and values as rows.

例如: 假设我有2条这样的记录-

For eg: Let say I have 2 records like this-

1. key1 -> value1, key2 -> value2, key3 -> value3 ....
2. key1 -> value11, key3 -> value13, key4 -> value14 ...

我希望输出数据框为

key1             key2                 key3             key4
value1           value2               value3            null
value11          null                 value13           value14

我该怎么做?

推荐答案

首先，我们需要创建一个id列，通过该列可以对您的数据进行分组，然后创建explode映射列A1，最后对您的数据进行整形df使用pivot():

First we need to create an id column by which we can group your data, then explode the map column A1, and finally reshape your df using pivot():

import org.apache.spark.sql.functions.{monotonically_increasing_id, explode, first}

df.withColumn("id", (monotonically_increasing_id()))
  .select($"id", explode($"A1"))
  .groupBy("id")
  .pivot("key")
  .agg(first("value")).show()
+---+-------+------+-------+-------+
| id|   key1|  key2|   key3|   key4|
+---+-------+------+-------+-------+
|  0| value1|value2| value3|   null|
|  1|value11|  null|value13|value14|
+---+-------+------+-------+-------+

这篇关于从哈希映射创建数据帧，其中键作为列名称，值作为Spark中的行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！