numpy数组的Python内存使用 [英] Python memory usage of numpy arrays

问题描述

我正在使用 python 来分析一些大文件，但遇到了内存问题，所以我一直在使用 sys.getsizeof() 来尝试跟踪使用情况，但是 numpy 数组的行为很奇怪.这是一个涉及我必须打开的反照率地图的示例:

<预><代码>>>>将 numpy 导入为 np>>>导入结构>>>从 sys 导入 getsizeof>>>f = open('Albedo_map.assoc', 'rb')>>>获取大小(f)144>>>反照率 = struct.unpack('%df' % (7200*3600), f.read(7200*3600*4))>>>getsizeof(反照率)207360056>>>反照率 = np.array(反照率).reshape(3600,7200)>>>getsizeof(反照率)80

数据仍然存在，但对象的大小，即 3600x7200 像素地图，已从约 200 Mb 变为 80 字节.我希望我的内存问题已经结束，只是将所有内容都转换为 numpy 数组，但我觉得这种行为，如果属实，会在某种程度上违反某些信息论或热力学定律，或者其他什么，所以我倾向于相信 getsizeof() 不适用于 numpy 数组.有什么想法吗?

解决方案

您可以使用 array.nbytes 用于 numpy 数组，例如:

<预><代码>>>>将 numpy 导入为 np>>>从 sys 导入 getsizeof>>>a = [0] * 1024>>>b = np.array(a)>>>getsizeof(a)8264>>>b.nbytes8192

I'm using python to analyse some large files and I'm running into memory issues, so I've been using sys.getsizeof() to try and keep track of the usage, but it's behaviour with numpy arrays is bizarre. Here's an example involving a map of albedos that I'm having to open:

>>> import numpy as np
>>> import struct
>>> from sys import getsizeof
>>> f = open('Albedo_map.assoc', 'rb')
>>> getsizeof(f)
144
>>> albedo = struct.unpack('%df' % (7200*3600), f.read(7200*3600*4))
>>> getsizeof(albedo)
207360056
>>> albedo = np.array(albedo).reshape(3600,7200)
>>> getsizeof(albedo)
80

Well the data's still there, but the size of the object, a 3600x7200 pixel map, has gone from ~200 Mb to 80 bytes. I'd like to hope that my memory issues are over and just convert everything to numpy arrays, but I feel that this behaviour, if true, would in some way violate some law of information theory or thermodynamics, or something, so I'm inclined to believe that getsizeof() doesn't work with numpy arrays. Any ideas?

解决方案

You can use array.nbytes for numpy arrays, for example:

>>> import numpy as np
>>> from sys import getsizeof
>>> a = [0] * 1024
>>> b = np.array(a)
>>> getsizeof(a)
8264
>>> b.nbytes
8192

这篇关于numpy数组的Python内存使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！