替换数据框中的重复列 [英] Replace duplicate columns in data frame
问题描述
我在 pyspark
中有一个数据框
.该数据框包含一些带有特殊字符的列.
I have a data frame
in pyspark
. This data frame has say some columns with special characters.
cols = df.schema.names
cols
['abc test', 'test*abc', 'eng)test', 'abc_&test']
reps = ((' ', '_&'), ('(', '*_'), (')', '_*'), ('{', '#_'), ('}', '_#'), (';', '_##'), ('.', '_$'), (',', '_$$'), ('=', '_**'))
def col_rename(x):
new_cols = reduce(lambda a, kv: a.replace(*kv), reps, x)
for i in cols:
df = df.withColumnRenamed(i, col_rename(cols, i))
return df
现在,我想查看是否在替换列名称中的特殊字符后是否存在任何重复的列.如我们所见,在new_cols abc_& test
Now I want to see if after replacing the special characters in the column names if there are any duplicate columns.
As we can see there is a duplicate of columns in the new_cols abc_&test
在发生这种情况时,我想返回额外的 _
下划线
.
I want to return extra _
underscore
when this happens.
我的new_cols应该像下面的
My new_cols shoul be like below
['abc__&test', 'test*_abc', 'eng_*test', 'abc_&test']
我如何实现自己想要的?
How can I achieve what I want?
推荐答案
首先,您需要更改
First of all you would need to change the column names as defined in
reps = [(' ', '_&'), ('(', '*_'), (')', '_*'), ('{', '#_'), ('}', '_#'), (';', '_##'), ('.', '_$'), (',', '_$$'), ('=', '_**')]
可以通过创建新列表
replacedCols = []
for col in cols:
for x in reps:
col = col.replace(x[0], x[1])
replacedCols.append(col)
现在,我想查看是否在替换列名称中的特殊字符后是否存在任何重复的列.发生这种情况时,我想返回多余的_下划线.
Now I want to see if after replacing the special characters in the column names if there are any duplicate columns. I want to return extra _ underscore when this happens.
您可以通过检查 replacedCols
数组
You can do that by checking for each column names in the replacedCols
array
checkCols = replacedCols[:]
for index, col in enumerate(replacedCols):
checkCols[index] = ''
replacedCols[index]
if col in checkCols:
replacedCols[index] = col.replace('_', '__')
到此您就完成了.最后一步是重命名列
Thus you are done. Final step is to rename the columns
for index, col in enumerate(cols):
df = df.withColumnRenamed(col, replacedCols[index])
df.show(truncate=False)
您应该拥有
+----------+--------+---------+---------+
|abc__&test|test*abc|eng_*test|abc_&test|
+----------+--------+---------+---------+
我希望这会有所帮助.编码愉快.
I hope this helps. Happy coding.
这篇关于替换数据框中的重复列的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!