如何使用pyspark更新结构嵌套列中的值 [英] How to update a value in the nested column of struct using pyspark
本文介绍了如何使用pyspark更新结构嵌套列中的值的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我尝试做的很简单 - 更新嵌套列的值;但是,我不知道如何
I try to do very simple - update a value of a nested column;however, I cannot figure out how
环境:
- Apache Spark 2.4.5
- Databricks 6.4
- Python 3.7
dataDF = [
(('Jon','','Smith'),'1580-01-06','M',3000)
]
schema = StructType([
StructField('name', StructType([
StructField('firstname', StringType(), True),
StructField('middlename', StringType(), True),
StructField('lastname', StringType(), True)
])),
StructField('dob', StringType(), True),
StructField('gender', StringType(), True),
StructField('gender', IntegerType(), True)
])
df = spark.createDataFrame(data = dataDF, schema = schema)
df = df.withColumn("name.firstname", lit('John'))
df.printSchema()
df.show()
#Results
#I get a new column instead of update
root
|-- name: struct (nullable = true)
| |-- firstname: string (nullable = true)
| |-- middlename: string (nullable = true)
| |-- lastname: string (nullable = true)
|-- dob: string (nullable = true)
|-- gender: string (nullable = true)
|-- gender: integer (nullable = true)
|-- name.firstname: string (nullable = false)
+--------------+----------+------+------+--------------+
| name| dob|gender|gender|name.firstname|
+--------------+----------+------+------+--------------+
|[Jon, , Smith]|1580-01-06| M| 3000| John|
+--------------+----------+------+------+--------------+
推荐答案
需要稍微纠结一下的栏目如下:
Need to wrangle with the column a bit as below:
import pyspark.sql.functions as F
df2 = df.select('*', 'name.*') \
.withColumn('firstname', F.lit('newname')) \
.withColumn('name', F.struct(*[F.col(col) for col in df.select('name.*').columns])) \
.drop(*df.select('name.*').columns)
df2.show()
+------------------+----------+------+------+
| name| dob|gender|gender|
+------------------+----------+------+------+
|[newname, , Smith]|1580-01-06| M| 3000|
+------------------+----------+------+------+
这篇关于如何使用pyspark更新结构嵌套列中的值的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文