如何根据第一列的内容拆分一个巨大的csv文件? [英] How to split a huge csv file based on content of first column?

问题描述

• 我有250MB +的巨大csv文件要上传
• 文件格式为group_id, application_id, reading，数据看起来像

• I have a 250MB+ huge csv file to upload
• file format is group_id, application_id, reading and data could look like

1, a1, 0.1
1, a1, 0.2
1, a1, 0.4
1, a1, 0.3
1, a1, 0.0
1, a1, 0.9
2, b1, 0.1
2, b1, 0.2
2, b1, 0.4
2, b1, 0.3
2, b1, 0.0
2, b1, 0.9
.....
n, x, 0.3(lets say)  

• 我想基于group_id划分文件，因此输出应为n个文件，其中n=group_id
• I want to divide the file based on group_id, so output should be n files where n=group_id

输出

File 1

1, a1, 0.1
1, a1, 0.2
1, a1, 0.4
1, a1, 0.3
1, a1, 0.0
1, a1, 0.9

和

File2
2, b1, 0.1
2, b1, 0.2
2, b1, 0.4
2, b1, 0.3
2, b1, 0.0
2, b1, 0.9
.....

和

File n
n, x, 0.3(lets say)  

我该如何有效地做到这一点?

How can I do this effectively?

推荐答案

如果文件已经由group_id排序，则可以执行以下操作:

If the file is already sorted by group_id, you can do something like:

import csv
from itertools import groupby

for key, rows in groupby(csv.reader(open("foo.csv")),
                         lambda row: row[0]):
    with open("%s.txt" % key, "w") as output:
        for row in rows:
            output.write(",".join(row) + "\n")

这篇关于如何根据第一列的内容拆分一个巨大的csv文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！