Python进程仅使用Numpy阵列中的1.6 GB RAM Ubuntu 32位 [英] Python Process using only 1.6 GB RAM Ubuntu 32 bit in Numpy Array

问题描述

我有一个用于学习人工神经网络的程序，它需要一个2 numpy数组作为训练数据.我要使用的数据数组的大小约为300,000 x 400浮点数.我不能在这里使用分块，因为我正在使用的库(DeepLearningTutorials)将单个numpy数组用作训练数据.

I have a program for learning Artificial Neural Network and it takes a 2-d numpy array as training data. The size of the data array I want to use is around 300,000 x 400 floats. I can't use chunking here because the library I am using (DeepLearningTutorials) takes a single numpy array as training data.

通过此过程(我在系统监视器中检查了)，当RAM使用量约为1.6 Gb时，代码显示MemoryError，但我的总RAM为8 GB.另外，该系统是32位Ubuntu-12.04.

The code shows MemoryError when the RAM usage is around 1.6Gb by this process(I checked it in system monitor) but I have a total RAM of 8GB. Also, the system is Ubuntu-12.04 32-bit.

我检查了其他类似问题的答案，但在某处说没有什么比向python程序分配内存更复杂了，而在某处却不清楚如何增加进程内存.

I checked for the answers ofor other similar questions but somewhere it says that there is nothing like allocating memory to your python program and somewhere the answer is not clear as to how to increase the process memory.

一个有趣的事情是我在另一台机器上运行相同的代码，它可以采用几乎1,500,000 x 400浮点数的numpy数组，而不会出现任何问题.基本配置类似，不同之处在于另一台计算机是64位，而另一台计算机是32位.

One interesting thing is I am running the same code on a different machine and it can take a numpy array of almost 1,500,000 x 400 floats without any problem. The basic configurations are similar except that the other machine is 64-bit and this one is 32-bit.

请问有人可以给出理论上的答案，为什么这有很大的不同?或者这是我遇到问题的唯一原因吗?

Could someone please give some theoretical answer as to why there is so much difference in this or is this the only reason for my problem?

推荐答案

32位操作系统最多只能处理4GB的RAM，而64位OS可以利用更多的RAM(理论上为1,680万TB).由于您的操作系统是32位的，因此您的操作系统只能利用4gb，因此不会使用其他4gb.

A 32-bit OS can only address up to aroung 4gb of ram, while a 64-bit OS can take advantage of a lot more ram (theoretically 16.8 million terabytes). Since your OS is 32-bit, your OS can only take advantage of 4gb, so your other 4gb isn't used.

另一台64位计算机没有4gb的内存限制，因此可以利用其所有已安装的内存.

The other 64-bit machine doesn't have the 4gb ram limit, so it can take advantage of all of its installed ram.

这些限制来自以下事实:32位计算机只能存储32字节的内存地址(指针)，因此计算机可以识别2 ^ 32个不同的可能内存位置.同样，一台64位计算机可以标识2 ^ 64个不同的可能的内存位置，因此它可以寻址2 ^ 64个不同的字节.

These limits come from the fact that a 32-bit machine can only store memory address (pointers) of 32-bytes, so there are 2^32 different possible memory locations that the computer can identify. Similarly, a 64-bit machine can identify 2^64 different possible memory locations, so it can address 2^64 different bytes.

这篇关于Python进程仅使用Numpy阵列中的1.6 GB RAM Ubuntu 32位的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！