在 Numpy Array 中仅使用 1.6 GB RAM Ubuntu 32 位的 Python 进程 [英] Python Process using only 1.6 GB RAM Ubuntu 32 bit in Numpy Array

查看:34
本文介绍了在 Numpy Array 中仅使用 1.6 GB RAM Ubuntu 32 位的 Python 进程的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我有一个学习人工神经网络的程序,它需要一个二维 numpy 数组作为训练数据.我要使用的数据数组的大小约为 300,000 x 400 浮点数.我不能在这里使用分块,因为我使用的库 (DeepLearningTutorials) 将单个 numpy 数组作为训练数据.

I have a program for learning Artificial Neural Network and it takes a 2-d numpy array as training data. The size of the data array I want to use is around 300,000 x 400 floats. I can't use chunking here because the library I am using (DeepLearningTutorials) takes a single numpy array as training data.

当此进程的 RAM 使用量约为 1.6Gb 时,代码显示 MemoryError(我在系统监视器中检查过),但我的总 RAM 为 8GB.此外,系统是 Ubuntu-12.04 32 位.

The code shows MemoryError when the RAM usage is around 1.6Gb by this process(I checked it in system monitor) but I have a total RAM of 8GB. Also, the system is Ubuntu-12.04 32-bit.

我检查了或其他类似问题的答案,但在某处它说没有什么比为您的 python 程序分配内存更合适的了,并且在某处关于如何增加进程内存的答案并不清楚.

I checked for the answers ofor other similar questions but somewhere it says that there is nothing like allocating memory to your python program and somewhere the answer is not clear as to how to increase the process memory.

一件有趣的事情是我在另一台机器上运行相同的代码,它可以使用几乎 1,500,000 x 400 浮点数的 numpy 数组而没有任何问题.基本配置类似,只是另一台机器是64位的,这台机器是32位的.

One interesting thing is I am running the same code on a different machine and it can take a numpy array of almost 1,500,000 x 400 floats without any problem. The basic configurations are similar except that the other machine is 64-bit and this one is 32-bit.

有人可以给出一些理论上的答案来解释为什么这之间存在如此大的差异还是这是我的问题的唯一原因?

Could someone please give some theoretical answer as to why there is so much difference in this or is this the only reason for my problem?

推荐答案

32 位操作系统最多只能处理大约 4GB 的内存,而 64 位操作系统可以利用更多内存(理论上 1680 万)TB).由于您的操作系统是 32 位,您的操作系统只能利用 4gb,因此您的其他 4gb 没有被使用.

A 32-bit OS can only address up to aroung 4gb of ram, while a 64-bit OS can take advantage of a lot more ram (theoretically 16.8 million terabytes). Since your OS is 32-bit, your OS can only take advantage of 4gb, so your other 4gb isn't used.

另一台 64 位机器没有 4gb 内存限制,因此它可以利用其安装的所有内存.

The other 64-bit machine doesn't have the 4gb ram limit, so it can take advantage of all of its installed ram.

这些限制来自这样一个事实,即 32 位机器只能存储 32 字节的内存地址(指针),因此计算机可以识别 2^32 种不同的可能内存位置.同样,64 位机器可以识别 2^64 个不同的可能内存位置,因此它可以寻址 2^64 个不同的字节.

These limits come from the fact that a 32-bit machine can only store memory address (pointers) of 32-bytes, so there are 2^32 different possible memory locations that the computer can identify. Similarly, a 64-bit machine can identify 2^64 different possible memory locations, so it can address 2^64 different bytes.

这篇关于在 Numpy Array 中仅使用 1.6 GB RAM Ubuntu 32 位的 Python 进程的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆