PySpark将类型为“映射"的列转换为数据框中的多个列 [英] PySpark converting a column of type 'map' to multiple columns in a dataframe
本文介绍了PySpark将类型为“映射"的列转换为数据框中的多个列的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我有一个类型为map
的列Parameters
,格式为:
I have a column Parameters
of type map
of the form:
>>> from pyspark.sql import SQLContext
>>> sqlContext = SQLContext(sc)
>>> d = [{'Parameters': {'foo': '1', 'bar': '2', 'baz': 'aaa'}}]
>>> df = sqlContext.createDataFrame(d)
>>> df.collect()
[Row(Parameters={'foo': '1', 'bar': '2', 'baz': 'aaa'})]
输出
我想在pyspark中重塑它,以便所有键(foo
,bar
等)都是列,即:
Output
I want to reshape it in pyspark so that all the keys (foo
, bar
, etc.) are columns, namely:
[Row(foo='1', bar='2', baz='aaa')]
使用withColumn
可以正常工作:
(df
.withColumn('foo', df.Parameters['foo'])
.withColumn('bar', df.Parameters['bar'])
.withColumn('baz', df.Parameters['baz'])
.drop('Parameters')
).collect()
但是我需要一种不明确提及列名的解决方案,因为我有数十种.
But I need like a solution that doesn't explicitly mention the column names as I have dozens of them.
>>> df.printSchema()
root
|-- Parameters: map (nullable = true)
| |-- key: string
| |-- value: string (valueContainsNull = true)
推荐答案
由于MapType
的键不是架构的一部分,因此您必须首先收集这些键,例如:
Since keys of the MapType
are not a part of the schema you'll have to collect these first for example like this:
from pyspark.sql.functions import explode
keys = (df
.select(explode("Parameters"))
.select("key")
.distinct()
.rdd.flatMap(lambda x: x)
.collect())
有了这些后,剩下的就是简单的选择:
When you have this all what is left is simple select:
from pyspark.sql.functions import col
exprs = [col("Parameters").getItem(k).alias(k) for k in keys]
df.select(*exprs)
这篇关于PySpark将类型为“映射"的列转换为数据框中的多个列的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文