使用np.zeros时，Windows而不是MacOS上出现MemoryError [英] MemoryError on Windows but not MacOS when using np.zeros

问题描述

我想构建一个numpy数组，但是在具有16 GB RAM的Windows中出现MemoryError.要提供更多信息，详细信息如下:

I want to build a numpy array but I get MemoryError in windows that has 16 GB RAM. To give more information the details are as follows:

Python 2.7.13 |Anaconda custom (64-bit)| (default, May 11 2017, 13:17:26) [MSCv.1500 64 bit (AMD64)] on win32

xx = np.zeros((110000,80000,3))

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
MemoryError

但是，完全相同的代码在具有8 GB RAM的Mac计算机上运行.详细信息是:

However exactly the same code runs on mac computer with 8 GB RAM. The details are:

Python 2.7.12 |Anaconda 4.2.0 (x86_64) [GCC 4.2.1 (LLVM build ... )on darwin)]

xx = np.zeros((1100000,8000000,3)) doesn't give error. 

最后增加尺寸时，在Mac上出现错误，但错误如下:

And when I increase the size at last I get error on Mac however the error is different as follows:

xx = np.zeros((1100000,80000000,3))
python(713,0x7fffa76b33c0) malloc: *** mach_vm_map(size=211200000000000) 
failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
MemoryError

我该如何解决这个问题?

How could I deal with this problem?

推荐答案

您的数组太大，无法放入两个系统的物理RAM中:

Your array is too huge to fit in the physical RAM on both systems:

110000 * 80000 * 3 * 8 / 1024 / 1024 / 1024 = 196.[69] GB

但是，这取决于您的系统如何处理内存请求以及如何存储"内存.例如，系统可以简单地为您的进程保留"内存，但是将实际分配推迟到读取/修改内存为止.即使分配了内存，也可以压缩内存(如注释中所述，感谢@Martijn Pieters)，并且可以很好地压缩很多零-但是，一旦修改内存，压缩效率就会降低，并且使用更多的实际内存".

However it depends on your system how it deals with the memory-request and how it "stores" the memory. For example the system could simply "reserve" memory for your process but postpones the actual allocation until the memory is read/modified. And even if the memory is allocated the memory can be compressed (as noted in the comments, thanks @Martijn Pieters) and having lots of zeros can be compressed really well - however as soon as you modify the memory the compression will become less efficient and more "real memory" is used.

这意味着它在失败时取决于系统(如果您实际上对阵列执行某些操作，则两者最终都会失败).如果您请求的物理内存超过了您的要求，Windows会立即选择立即失败.在Mac上，您似乎必须修改足够"的值(另请参见有关为什么可以在352GB NumPy ndarray上使用此答案"的答案8GB内存的macOS计算机?" )，直到出现故障.

That means it's up to the system when it fails (both will eventually fail if you actually do something with the array). Windows in your case chooses to fail immediately when you request more physical memory than you have. On Mac it seems like you have to modify "enough" values (see also this answer on "Why a 352GB NumPy ndarray can be used on a 8GB memory macOS computer?") until it fails.

import numpy as np
arr = np.zeros((110000,80000,3))  # MemoryError on Windows
arr += 10                         # MemoryError on Mac

例如，您可以使用psutil来检查已使用的内存量(物理和虚拟):

You could use for example psutil to check the amount of memory used (physical and virtual):

import psutil
print(psutil.virtual_memory())  
# svmem(total=4170924032, available=1666629632, percent=60.0, used=2504294400, free=1666629632)
arr = np.zeros((10000, 10000))
print(psutil.virtual_memory())
# svmem(total=4170924032, available=1664675840, percent=60.1, used=2506248192, free=1664675840)
arr += 10
print(psutil.virtual_memory())
# svmem(total=4170924032, available=864059392, percent=79.3, used=3306864640, free=864059392)

因此，即使在Windows上，np.zeros也不会立即使用"物理内存，直到需要它为止.

So even on Windows the np.zeros doesn't immediately "use" the physical memory until it is needed.

我该如何解决这个问题?

How could I deal with this problem ?

最简单(但可能成本最高)的选择是购买(大量)更多的RAM.但是，您也可以尝试使用适合物理内存的块大小来分块"处理.最终，最好的解决方案是重新考虑您的方法/算法，以便您不需要那么多的内存.

The easiest (but probably most costly) option would be to buy (a lot) more RAM. But you could also try to "chunk" the processing with chunk sizes that fit into your physical memory. Ultimately the best solution would be to rethink your approach/algorithm so you don't need that much memory.

这篇关于使用np.zeros时，Windows而不是MacOS上出现MemoryError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！