无法分配具有形状和数据类型的数组 [英] Unable to allocate array with shape and data type

问题描述

我在Ubuntu 18上在numpy中分配大型数组时遇到了一个问题，而在MacOS上却没有遇到同样的问题.

I'm facing an issue with allocating huge arrays in numpy on Ubuntu 18 while not facing the same issue on MacOS.

我正在尝试为形状为(156816, 36, 53806)的numpy数组分配内存

I am trying to allocate memory for a numpy array with shape (156816, 36, 53806) with

np.zeros((156816, 36, 53806), dtype='uint8')

在Ubuntu操作系统上出现错误时

and while I'm getting an error on Ubuntu OS

>>> import numpy as np
>>> np.zeros((156816, 36, 53806), dtype='uint8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
numpy.core._exceptions.MemoryError: Unable to allocate array with shape (156816, 36, 53806) and data type uint8

我在MacOS上没有得到它:

I'm not getting it on MacOS:

>>> import numpy as np 
>>> np.zeros((156816, 36, 53806), dtype='uint8')
array([[[0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        ...,
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0]],

       [[0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        ...,
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0]],

       [[0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        ...,
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0]],

       ...,

       [[0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        ...,
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0]],

       [[0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        ...,
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0]],

       [[0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        ...,
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0],
        [0, 0, 0, ..., 0, 0, 0]]], dtype=uint8)

我在某处读到，np.zeros不应真正分配数组所需的全部内存，而只能分配非零元素.即使Ubuntu计算机具有64GB的内存，而我的MacBook Pro却只有16GB.

I've read somewhere that np.zeros shouldn't be really allocating the whole memory needed for the array, but only for the non-zero elements. Even though the Ubuntu machine has 64gb of memory, while my MacBook Pro has only 16gb.

版本:

Ubuntu
os -> ubuntu mate 18
python -> 3.6.8
numpy -> 1.17.0

mac
os -> 10.14.6
python -> 3.6.4
numpy -> 1.17.0

PS:在Google Colab上也失败

PS: also failed on Google Colab

推荐答案

这可能是由于系统的过量提交处理模式.

This is likely due to your system's overcommit handling mode.

在默认模式下，0

启发式过量使用处理.明显的地址空间过量使用被拒绝.用于典型的系统.它确保严重的野生分配失败，同时允许过量使用以减少交换使用.在此模式下，允许root分配更多的内存.这是默认设置.

Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default.

此处未对使用的确切启发式进行很好的解释，但这在 Linux over commit heuristic 和在此页面上.

The exact heuristic used is not well explained here, but this is discussed more on Linux over commit heuristic and on this page.

您可以通过运行来检查当前的过量使用模式

You can check your current overcommit mode by running

$ cat /proc/sys/vm/overcommit_memory
0

在这种情况下，您正在分配

In this case you're allocating

>>> 156816 * 36 * 53806 / 1024.0**3
282.8939827680588

〜282 GB，并且内核说的很清楚，我无法将这么多物理页面提交到该页面，并且它拒绝分配.

~282 GB, and the kernel is saying well obviously there's no way I'm going to be able to commit that many physical pages to this, and it refuses the allocation.

如果(以root用户身份)运行:

If (as root) you run:

$ echo 1 > /proc/sys/vm/overcommit_memory

这将启用始终过量使用"模式，并且您会发现，无论系统有多大(至少在64位内存寻址内)，系统都可以进行分配.

This will enable "always overcommit" mode, and you'll find that indeed the system will allow you to make the allocation no matter how large it is (within 64-bit memory addressing at least).

我自己在具有32 GB RAM的计算机上对此进行了测试.使用过量提交模式0时，我也得到了MemoryError，但是将其更改回1后，它可以正常工作:

I tested this myself on a machine with 32 GB of RAM. With overcommit mode 0 I also got a MemoryError, but after changing it back to 1 it works:

>>> import numpy as np
>>> a = np.zeros((156816, 36, 53806), dtype='uint8')
>>> a.nbytes
303755101056

然后，您可以继续写入数组中的任何位置，并且仅当您明确写入物理页面时，系统才会分配物理页面.因此，您可以谨慎地将其用于稀疏数组.

You can then go ahead and write to any location within the array, and the system will only allocate physical pages when you explicitly write to that page. So you can use this, with care, for sparse arrays.

这篇关于无法分配具有形状和数据类型的数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！