多少内存在numpy的阵列？ RAM是一个限制因素？ [英] How much memory in numpy array? Is RAM a limiting factor?

问题描述

我使用numpy的创建与长度为100的边立方体阵列，从而含有1万个条目总数。对于每个万个条目的，我插入一个100x100的矩阵，其项是由随机生成的数字。我使用下面的code这样做的：

I'm using numpy to create a cube array with sides of length 100, thus containing 1 million entries total. For each of the million entries, I am inserting a 100x100 matrix whose entries are comprised of randomly generated numbers. I am using the following code to do so:

import random
from numpy import *

cube = arange(1000000).reshape(100,100,100)

for element in cube.flat:
    matrix = arange(10000).reshape(100,100)
    for entry in matrix.flat:
        entry = random.random()*100
    element = matrix

我期待这需要一段时间，但与正在生成10个十亿的随机数，我不知道我的电脑甚至可以处理它。这样一种阵列就多少内存占用？将RAM是一个限制因素，即如果我的计算机没有足够的内存，可以把它不能实际生成的阵列？

I was expecting this to take a while, but with 10 billion random numbers being generated, I'm not sure my computer can even handle it. How much memory would such an array take up? Would RAM be a limiting factor, i.e. if my computer doesn't have enough RAM, could it fail to actually generate the array?

此外，如果有一个更有效的实现这个code，我会AP preciate提示：）

Also, if there is a more efficient to implement this code, I would appreciate tips :)

推荐答案

一对夫妇点：

• 在numpy的阵列内存的大小很容易计算。它只是倍的数据大小的元素的数量，加上一个小的恒定开销。例如，如果你的 cube.dtype 是的Int64 ，它共有1,000,000次的元素，它会要求百万* 64/8 = 800万字节（8MB）。


• 然而，随着@Gabe指出，100 * 100 *百万双打将需要约的 80 GB 。


• 这不会引起任何破发，每本身，但操作会因为所有的交换您的计算机将需要做的。


• 您的循环不会做你期望的。相反，在立方体，元素=矩阵替代的元素将简单地覆盖元素变量，留下立方体不变。这同样适用于在项= random.rand（）* 100 。


• 相反，请参见：<一href=\"http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html#modifying-array-values\">http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html#modifying-array-values

• The size in memory of numpy arrays is easy to calculate. It's simply the number of elements times the data size, plus a small constant overhead. For example, if your cube.dtype is int64, and it has 1,000,000 elements, it will require 1000000 * 64 / 8 = 8,000,000 bytes (8Mb).
• However, as @Gabe notes, 100 * 100 * 1,000,000 doubles will require about 80 Gb.
• This will not cause anything to "break", per-se, but operations will be ridiculously slow because of all the swapping your computer will need to do.
• Your loops will not do what you expect. Instead of replacing the element in cube, element = matrix will simply overwrite the element variable, leaving the cube unchanged. The same goes for the entry = random.rand() * 100.
• Instead, see: http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html#modifying-array-values

这篇关于多少内存在numpy的阵列？ RAM是一个限制因素？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！