Mac OS X和Linux上的Numpy和内存分配 [英] Numpy and memory allocation on Mac OS X vs. Linux

问题描述

我使用numpy使用64位Python加载大型矩阵.

I use numpy to load a large matrix using 64bit Python.

它在具有8GB内存的Macbook Pro上运行良好.

It works fine on Macbook Pro with 8GB memory.

>>> from sklearn.preprocessing import MultiLabelBinarizer
>>> mb = MultiLabelBinarizer()
>>> matrix = mb.fit_transform(questions_topics)
>>> sys.getsizeof(matrix) 
47975472376
>>> matrix.shape
(2999967, 1999)

但是在具有16GB内存和10GB交换空间的Ubuntu Google VM实例上，它会提高MemoryError.

But it raises MemoryError on Ubuntu Google VM instance with 16GB memory and 10GB swap.

>>> y = mb.fit_transform(questions_topics)
/home/Liwink/anaconda3/lib/python3.5/site-packages/scipy/sparse/base.py in _process_toarray_args(self, order, out)
1037             return out
1038         else:
-> 1039             return np.zeros(self.shape, dtype=self.dtype, order=order)
1040
1041     def __numpy_ufunc__(self, func, method, pos, inputs, **kwargs):
MemoryError:

在Mac OS上加载矩阵时，它需要50G VIRT.

When the matrix is loaded on Mac OS, it takes 50G VIRT.

我有两个问题:

1. 存储在内存或磁盘中的矩阵(约50GB)在哪里?
2. 如何在VM上加载此矩阵?

推荐答案

感谢@ juanpa.arrivillaga，我通过

Thanks @juanpa.arrivillaga, I solved the problem by increasing the swap.

但是它仍然不是完美的，因为在Ubuntu上它将首先使用内存，而在Mac OS上它将大量节省"内存.

But it is still not perfect, since on Ubuntu it will use up the memory first but on Mac OS it "saves" memory a lot.

在Ubuntu上:

On Ubuntu:

在Mac OS上:

On Mac OS:

在Ubuntu上，它比在Mac OS上使用更多的RES.

On Ubuntu, it uses much more RES than it on Mac OS.

如何在Ubuntu上保存内存?

How can I save the memory on Ubuntu?

这篇关于Mac OS X和Linux上的Numpy和内存分配的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！