Numpy 内存错误创建巨大的矩阵 [英] Numpy memory error creating huge matrix
问题描述
我正在使用 numpy 并尝试创建一个巨大的矩阵.执行此操作时，我收到内存错误
I am using numpy and trying to create a huge matrix. While doing this, I receive a memory error
因为矩阵并不重要，我将只展示如何轻松重现错误的方法.
Because the matrix is not important, I will just show the way how to easily reproduce the error.
a = 10000000000
data = np.array([float('nan')] * a)
毫不奇怪，这会引发我MemoryError
我想说两件事:
- 我真的需要创建和使用一个大矩阵
- 我想我有足够的 RAM 来处理这个矩阵(我有 24 Gb 或 RAM)
有没有一种简单的方法可以在 numpy 中处理大矩阵?
Is there an easy way to handle big matrices in numpy?
为了安全起见，我以前读过这些帖子(听起来很相似):
Just to be on the safe side, I previously read these posts (which sounds similar):
P.S.显然我在数字的乘除上有一些问题，这让我觉得我有足够的记忆.所以我觉得是时候去睡觉了，复习数学，也许是买一些内存.
P.S. apparently I have some problems with multiplication and division of numbers, which made me think that I have enough memory. So I think it is time for me to go to sleep, review math and may be to buy some memory.
可能在这段时间里，一些天才可能会想出如何仅使用 24 Gb 的 Ram 来实际创建这个矩阵的想法.
May be during this time some genius might come up with idea how to actually create this matrix using only 24 Gb of Ram.
为什么我需要这么大的矩阵我不会对这个矩阵做任何操作.我需要做的就是将它保存到 pytables 中.
Why I need this big matrix I am not going to do any manipulations with this matrix. All I need to do with it is to save it into pytables.
推荐答案
假设每个浮点数是 4 个字节，你就会有
Assuming each floating point number is 4 bytes each, you'd have
(10000000000 * 4) /(2**30.0) = 37.25290298461914
或者您需要在内存中存储 37.5 GB.所以我认为 24GB 的 RAM 是不够的.
Or 37.5 gigabytes you need to store in memory. So I don't think 24gb of RAM is enough.
这篇关于Numpy 内存错误创建巨大的矩阵的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
