替换数据框中的重复列 [英] Replace duplicate columns in data frame

问题描述

我在 pyspark 中有一个数据框.该数据框包含一些带有特殊字符的列.

I have a data frame in pyspark. This data frame has say some columns with special characters.

cols = df.schema.names

cols
['abc test', 'test*abc', 'eng)test', 'abc_&test']

reps = ((' ', '_&'), ('(', '*_'), (')', '_*'), ('{', '#_'), ('}', '_#'), (';', '_##'), ('.', '_$'), (',', '_$$'), ('=', '_**'))

def col_rename(x):
    new_cols = reduce(lambda a, kv: a.replace(*kv), reps, x)

for i in cols:
    df = df.withColumnRenamed(i, col_rename(cols, i))
return df

现在，我想查看是否在替换列名称中的特殊字符后是否存在任何重复的列.如我们所见，在new_cols abc_& test

Now I want to see if after replacing the special characters in the column names if there are any duplicate columns. As we can see there is a duplicate of columns in the new_cols abc_&test

在发生这种情况时，我想返回额外的 _ 下划线.

I want to return extra _ underscore when this happens.

我的new_cols应该像下面的

My new_cols shoul be like below

['abc__&test', 'test*_abc', 'eng_*test', 'abc_&test']

我如何实现自己想要的?

How can I achieve what I want?

推荐答案

首先，您需要更改

First of all you would need to change the column names as defined in

reps = [(' ', '_&'), ('(', '*_'), (')', '_*'), ('{', '#_'), ('}', '_#'), (';', '_##'), ('.', '_$'), (',', '_$$'), ('=', '_**')]

可以通过创建新列表

replacedCols = []
for col in cols:
    for x in reps:
        col = col.replace(x[0], x[1])
    replacedCols.append(col)

现在，我想查看是否在替换列名称中的特殊字符后是否存在任何重复的列.发生这种情况时，我想返回多余的_下划线.

Now I want to see if after replacing the special characters in the column names if there are any duplicate columns. I want to return extra _ underscore when this happens.

您可以通过检查 replacedCols 数组

You can do that by checking for each column names in the replacedCols array

checkCols = replacedCols[:]
for index, col in enumerate(replacedCols):
    checkCols[index] = ''
    replacedCols[index]
    if col in checkCols:
        replacedCols[index] = col.replace('_', '__')

到此您就完成了.最后一步是重命名列

Thus you are done. Final step is to rename the columns

for index, col in enumerate(cols):
    df = df.withColumnRenamed(col, replacedCols[index])

df.show(truncate=False)

您应该拥有

+----------+--------+---------+---------+
|abc__&test|test*abc|eng_*test|abc_&test|
+----------+--------+---------+---------+

我希望这会有所帮助.编码愉快.

I hope this helps. Happy coding.

这篇关于替换数据框中的重复列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！