大数组的块状直方图 [英] Numpy histogram of large arrays

问题描述

我有一堆csv数据集，每个数据集的大小约为10Gb.我想从他们的列中生成直方图.但是似乎在numpy中执行此操作的唯一方法是首先将整个列加载到numpy数组中，然后在该数组上调用numpy.histogram.这会消耗不必要的内存.

I have a bunch of csv datasets, about 10Gb in size each. I'd like to generate histograms from their columns. But it seems like the only way to do this in numpy is to first load the entire column into a numpy array and then call numpy.histogram on that array. This consumes an unnecessary amount of memory.

numpy是否支持在线分箱?我希望在我的csv上逐行进行迭代，并在读取它们时将值归类.这样，任何时候最多只有一行存储在内存中.

Does numpy support online binning? I'm hoping for something that iterates over my csv line by line and bins values as it reads them. This way at most one line is in memory at any one time.

不难推出自己的产品，但想知道是否有人已经发明了这种车轮.

Wouldn't be hard to roll my own, but wondering if someone already invented this wheel.

推荐答案

正如您所说，推出自己的产品并不难.您需要自己设置垃圾箱，并在遍历文件时重复使用它们.以下应该是一个不错的起点:

As you said, it's not that hard to roll your own. You'll need to set up the bins yourself and reuse them as you iterate over the file. The following ought to be a decent starting point:

import numpy as np
datamin = -5
datamax = 5
numbins = 20
mybins = np.linspace(datamin, datamax, numbins)
myhist = np.zeros(numbins-1, dtype='int32')
for i in range(100):
    d = np.random.randn(1000,1)
    htemp, jnk = np.histogram(d, mybins)
    myhist += htemp

我猜想对于如此大的文件，性能将是一个问题，并且在每一行上调用直方图的开销可能太慢. @doug关于生成器的建议似乎是解决该问题的一种好方法问题.

I'm guessing performance will be an issue with such large files, and the overhead of calling histogram on each line might be too slow. @doug's suggestion of a generator seems like a good way to address that problem.

这篇关于大数组的块状直方图的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！