为什么从文件中读取numpy的narray消耗这么多的内存？ [英] why numpy narray read from file consumes so much memory?

问题描述

该文件包含200万行：
每一行包含列208，用逗号分隔，如：

<$p$p>0.0863314058048,0.0208767447842,0.03358010485,0.0,1.0,0.0,0.314285714286,0.336293217457,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0

程序读取这个文件到numpy的narray，我预期将消耗约（2000000 * 208 * 8B）= 3.2GB 内存。
然而，当程序读取这个文件，我发现程序消耗约20GB的内存。

我感到困惑，为什么我的程序会消耗这么多的内存没有达到预期？

解决方案

我使用numpy的1.9.0和 np.loadtxt的内存inneficiency（）和 np.genfromtxt（）似乎直接相关它们基于临时列表中的事实来存储数据：

• 请参见这里 np.loadtxt（）


• 和这里 np.genfromtxt（）

通过事先知道形状您的数组的你可以把文件阅读器会消耗内存非常接近理论量的内存（3.2 GB的金额的这种情况下），通过使用存储数据的相应的 DTYPE ：

 高清read_large_txt（路径分隔符=无，DTYPE =无）：
    开放（路径）为f：
        NROWS = SUM（1为F线）
        f.seek（0）
        NCOLS = LEN（f.next（）。拆分（分隔符））
        OUT = np.empty（（NROWS，NCOLS），DTYPE DTYPE =）
        f.seek（0）
        因为我在枚举线（F）：
            出[I] = line.split（分隔符）
    返回了
 

the file contains 2000000 rows: each row contains 208 columns, separated by comma, like this:

0.0863314058048,0.0208767447842,0.03358010485,0.0,1.0,0.0,0.314285714286,0.336293217457,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0,0,0,0

The program read this file to a numpy narray, I expected it will consume about (2000000 * 208 * 8B) = 3.2GB memory. However, when the program read this file, I found the program consumes about 20GB memory.

I am confused about why my program consumes so much memory that do not meet expectation?

解决方案

I'm using Numpy 1.9.0 and the memory inneficiency of np.loadtxt() and np.genfromtxt() seems to be directly related to the fact they are based on temporary lists to store the data:

• see here for np.loadtxt()
• and here for np.genfromtxt()

By knowing beforehand the shape of your array you can think of a file reader that will consume an amount of memory very close to the theoretical amount of memory (3.2 GB for this case), by storing the data using the corresponding dtype:

def read_large_txt(path, delimiter=None, dtype=None):
    with open(path) as f:
        nrows = sum(1 for line in f)
        f.seek(0)
        ncols = len(f.next().split(delimiter))
        out = np.empty((nrows, ncols), dtype=dtype)
        f.seek(0)
        for i, line in enumerate(f):
            out[i] = line.split(delimiter)
    return out

这篇关于为什么从文件中读取numpy的narray消耗这么多的内存？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！