如何转置/枢轴csv文件与python *，而无需将整个文件加载到内存？ [英] How do I transpose/pivot a csv file with python *without* loading the whole file into memory?

问题描述

对于我的一个数据分析管道，我最终生成了大量的CSV文件。我想转移他们，连接他们，并重新调换他们。但是，数据量很大，因此将其全部加载到内存中是不切实际的。

For one of my data analysis pipelines, I end up generating a lot of individual CSV files. I would like to transpose them, concatenate them, and transpose them again. However, the amount of data is large, so loading it all into memory is not practical.

推荐答案

/ p>

Use a generator, e.g.

from itertools import izip

file1 = open("test", "r")
file2 = open("test2", "r")

def lazy(file):
    for line in file:
        #do something with the line
        yield line

for lines in izip(lazy(file1), lazy(file2)):
    print lines

http://wiki.python.org/moin/Generators

编辑：您可以使用CSV模块来解析它，我也意识到文件对象的readlines（）方法不是惰性，以文件模式使用for行。

You can use the CSV module to parse it, also I realized that the readlines() method of file objects isn't lazy, so you have to use the for line in file pattern.

这篇关于如何转置/枢轴csv文件与python *，而无需将整个文件加载到内存？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！