Numpy内存错误创建了巨大的矩阵 [英] Numpy memory error creating huge matrix

问题描述

我正在使用numpy并尝试创建一个巨大的矩阵. 这样做时，我收到内存错误

I am using numpy and trying to create a huge matrix. While doing this, I receive a memory error

由于矩阵并不重要，因此我将向您展示如何轻松重现错误的方法.

Because the matrix is not important, I will just show the way how to easily reproduce the error.

a = 10000000000
data = np.array([float('nan')] * a)

毫不奇怪，这使我MemoryError

我想讲两件事:

1. 我真的需要创建并使用一个大矩阵
2. 我认为我有足够的RAM来处理此矩阵(我有24 Gb或RAM)

是否有一种简单的方法来处理numpy中的大型矩阵?

Is there an easy way to handle big matrices in numpy?

为了安全起见，我之前阅读过这些帖子(听起来很相似):

Just to be on the safe side, I previously read these posts (which sounds similar):

使用Python和NumPy的大型矩阵

Python/Numpy MemoryError

处理非常大的数据集在python中-内存错误

P.S.显然，我在数字的乘法和除法方面存在一些问题，这使我认为我有足够的记忆力.因此，我认为该是时候该睡觉了，复习数学，也许是买些记忆.

P.S. apparently I have some problems with multiplication and division of numbers, which made me think that I have enough memory. So I think it is time for me to go to sleep, review math and may be to buy some memory.

可能是在这段时间里，一些天才提出了如何仅使用24 Gb Ram实际创建此矩阵的想法.

May be during this time some genius might come up with idea how to actually create this matrix using only 24 Gb of Ram.

为什么我需要这个大矩阵 我不会对该矩阵进行任何操作.我需要做的就是将其保存到 pytables 中.

Why I need this big matrix I am not going to do any manipulations with this matrix. All I need to do with it is to save it into pytables.

推荐答案

假设每个浮点数均为4个字节，则您将拥有

Assuming each floating point number is 4 bytes each, you'd have

(10000000000 * 4) /(2**30.0) = 37.25290298461914

您需要在内存中存储37.5 GB.因此，我认为24gb的RAM不够.

Or 37.5 gigabytes you need to store in memory. So I don't think 24gb of RAM is enough.

这篇关于Numpy内存错误创建了巨大的矩阵的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！