内存在Windows 7蟒蛇32 python.exe - numpy的仅使用可用内存的一半？ [英] Memory for python.exe on Windows 7 python 32 - Numpy uses half of the available memory only?

问题描述

我加载大型H5文件到内存中使用 numpy的 ndarray 的。我读了我的系统（Win 7的教授，6 GB RAM）应该允许python.exe使用的物理内存的2 GB。

I'm loading large h5 files into memory using numpy ndarray's. I read that my system (Win 7 prof., 6 GB RAM) is supposed to allow python.exe to use about 2 GB of physical memory.

但是我收到一个的MemoryError 已略低于1 GB。更奇怪的这个下限，似乎只适用于 numpy的阵列的而不是为列表。

However I'm getting a MemoryError already just shy of 1 GB. Even stranger this lower limit seems to only apply for numpy array's but not for a list.

我用下面的函数找到的<一个测试我的内存消耗href=\"http://fa.bianp.net/blog/2013/different-ways-to-get-memory-consumption-or-lessons-learned-from-memory_profiler/\"相对=nofollow>这里：

I've tested my memory consumption using the following function found here:

import psutil
import gc
import os
import numpy as np
from matplotlib.pyplot import pause

def memory_usage_psutil():
    # return the memory usage in MB
    process = psutil.Process(os.getpid())
    mem = process.get_memory_info()[0]/float(2**20)
    return mem

测试1：作为一个普通的列表测试的内存限制。

Test 1: Testing memory limits for an ordinary list

print 'Memory - %d MB' %memory_usage_psutil() # prints memory usage after imports
a = []
while 1:
    try:
        a.append([x*2000 for x in xrange(10000)])
    except MemoryError:
        print 'Memory - %d MB' %memory_usage_psutil()
        a = []
        print 'Memory - %d MB' %memory_usage_psutil()
        print 'run garbage collector: collected %d objects.' %gc.collect()
        print 'Memory - %d MB\n\n' %memory_usage_psutil()
        break

测试1打印：

Memory - 39 MB
Memory - 1947 MB
Memory - 1516 MB
run garbage collector: collected 0 objects.
Memory - 49 MB

测试2：创建一批大型 np.array 的

shape = (5500,5500)
names = ['b', 'c', 'd', 'g', 'h']

try:
    for n in names:
        globals()[n] = np.ones(shape, dtype='float64')
        print 'created variable %s with %0.2f MB'\
        %(n,(globals()[n].nbytes/2.**20))
except MemoryError:
    print 'MemoryError, Memory - %d MB. Deleting files..'\
    %memory_usage_psutil()
    pause(2)
    # Just added the pause here to be able to observe
    # the spike of memory in the Windows task manager.
    for n in names:
        globals()[n] = []
    print 'Memory - %d MB' %memory_usage_psutil()
    print 'run garbage collector: collected %d objects.' %gc.collect()
    print 'Memory - %d MB' %memory_usage_psutil()

2测试打印：

Memory - 39 MB
created variable b with 230.79 MB
created variable c with 230.79 MB
created variable d with 230.79 MB
created variable g with 230.79 MB
MemoryError, Memory - 964 MB. Deleting files..
Memory - 39 MB
run garbage collector: collected 0 objects.
Memory - 39 MB

我的问题：为什么我收到了的MemoryError 之前，我甚至接近2GB限制，以及为什么会出现在内存限制的差异列表和 np.array 或我缺少什么？
我使用python 2.7和1.7.1 numpy的

My question: Why do I get a MemoryError before I'm even close to the 2GB limit and why is there a difference in memory limits for a list and np.array respectively or what am I missing? I'm using python 2.7 and numpy 1.7.1

推荐答案

这是可能发生，因为numpy的阵列使用一些C阵列库（速度），即某个地方调用一个malloc。然后，这会失败，因为它不能分配的内存的连续的1GB。我进一步猜测，Python列表作为一个链表实现的，因此，需要对列表中的内存不一定是连续的。因此，如果你有足够的可用内存，但它是支离破碎的，你的数组的malloc会失败，但你链接的列表将允许您使用所有的不连续件。

This is probably happening because numpy array is using some C array library (for speed), that is somewhere calling a malloc. This then fails because it cannot allocate a contiguous 1GB of memory. I am further guessing that Python lists are implemented as a linked list, thus the memory needed for a list need not be contiguous. Hence, if you have enough memory available but it is fragmented, your array malloc would fail but your linked list would allow you to use all of the noncontiguous pieces.

这篇关于内存在Windows 7蟒蛇32 python.exe - numpy的仅使用可用内存的一半？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！