如何相对于其他数据框更改数据框的列名 [英] How to change column name of a dataframe with respect to other dataframe
问题描述
我需要使用 pyspark
I have a requirement to change column name of a dataframe df
with respect to other dataframe df_col
using pyspark
df
+----+---+----+----+
|code| id|name|work|
+----+---+----+----+
| ASD|101|John| DEV|
| klj|102| ben|prod|
+----+---+----+----+
df_col
+-----------+-----------+
|col_current|col_updated|
+-----------+-----------+
| id| Row_id|
| name| Name|
| code| Row_code|
| Work| Work_Code|
+-----------+-----------+
如果 df 列与 col_current 匹配,则 df 列应替换为 col_updated.例如:如果 df.id 与 df.col_current 匹配,则 df.id 应替换为 Row_id.
if df column matches col_current, df column should replace with col_updated. ex: if df.id matches df.col_current, df.id should replace with Row_id.
预期输出
Row_id,Name,Row_code,Work_code
101,John,ASD,DEV
102,ben,klj,prod
注意:我希望这个过程是动态的.
Note: I want this process to be dynamic.
推荐答案
只需将 df_col
收集为字典:
Just collect the df_col
as dictionary:
df = spark.createDataFrame(
[("ASD", "101" "John", "DEV"), ("klj","102", "ben", "prod")],
("code", "id", "name", "work")
)
df_col = spark.createDataFrame(
[("id", "Row_id"), ("name", "Name"), ("code", "Row_code"), ("Work", "Work_Code")],
("col_current", "col_updated")
)
name_dict = df_col.rdd.collectAsMap()
并使用 select
与列表理解:
and use select
with list comprehension:
df.select([df[c].alias(name_dict.get(c, c)) for c in df.columns]).printSchema()
# root
# |-- Row_code: string (nullable = true)
# |-- Row_id: string (nullable = true)
# |-- Name: string (nullable = true)
# |-- work: string (nullable = true)
其中 name_dict
是标准 Python 字典:
where name_dict
is standard Python dictionary:
{'Work': 'Work_Code', 'code': 'Row_code', 'id': 'Row_id', 'name': 'Name'}
name_dict.get(c, c)
获取新名称、给定的当前名称或当前名称(如果不匹配):
name_dict.get(c, c)
gets new name, given current name, or current name if no match:
name_dict.get("code", "code")
# 'Row_code'
name_dict.get("work", "work") # Case sensitive
# 'work'
和 alias
只是将列 (df[col]
) 重命名为从 name_dict.get
返回的名称.
and alias
just renames column (df[col]
) to name returned from name_dict.get
.
这篇关于如何相对于其他数据框更改数据框的列名的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!