Python:删除重复的CSV条目 [英] Python: Removing duplicate CSV entries

问题描述

我有一个包含多个条目的CSV文件.范例csv:

I have a CSV file with multiple entries. Example csv:

user, phone, email
joe, 123, [e-mail]
mary, 456, [e-mail]
ed, 123, [e-mail]

我正在尝试通过CSV中的特定列删除重复项，但是使用下面的代码，我得到列表索引超出范围".我认为通过将row[1]与newrows[1]进行比较，我会找到所有重复项，并且只重写file2.csv中的唯一条目.但这不起作用，我也不明白为什么.

I'm trying to remove the duplicates by a specific column in the CSV however with the code below I'm getting an "list index out of range". I thought by comparing row[1] with newrows[1] I would find all duplicates and only rewrite the unique entries in file2.csv. This doesn't work though and I can't understand why.

f1 = csv.reader(open('file1.csv', 'rb'))
    newrows = []
    for row in f1:
        if row[1] not in newrows[1]:
            newrows.append(row)
    writer = csv.writer(open("file2.csv", "wb"))
    writer.writerows(newrows)

我的最终结果是拥有一个维护文件顺序的列表(set将无法正常工作...对吗?)，其外观应如下所示:

My end result is to have a list that maintains the sequence of the file (set won't work...right?) which should look like this:

user, phone, email
joe, 123, [e-mail]
mary, 456, [e-mail]

推荐答案

row[1]引用当前行(电话)的第二列.一切都很好.

row[1] refers to the second column in the current row (phone). That's all well in good.

但是，您newrows.append(row)将整行添加到列表中.

However, you newrows.append(row) add the entire row to the list.

当您检查row[1] in newrows时，您正在对照完整行列表检查单个电话号码.但这不是您想要的.您只需要检查一个列表或一组电话号码.为此，您可能想要跟踪行和一组观察到的电话号码.

When you check row[1] in newrows you are checking the individual phone number against a list of complete rows. But that's not what you want to do. You need to check against a list or set of just phone numbers. For that, you probably want to keep track of the rows and a set of the observed phone numbers.

类似的东西:

f1 = csv.reader(open('file1.csv', 'rb'))
writer = csv.writer(open("file2.csv", "wb"))
phone_numbers = set()
for row in f1:
    if row[1] not in phone_numbers:
        writer.writerow(row)
        phone_numbers.add( row[1] )

这篇关于Python:删除重复的CSV条目的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！