Python在64位窗口上的32位内存限制 [英] Python 32-bit memory limits on 64bit windows
问题描述
我在一台装有8GB内存的Windows 7 64位机器上运行一个32位的Python程序。
这些程序读取了5,118个压缩的numpy文件（npz）。
Windows报告这些文件占用磁盘1.98 GB
每个npz文件包含两段数据：
'arr_0'的类型是np .float32和
'arr_1'是np.uint8的类型。
python脚本读取每个文件将其数据附加到两个列表中，然后关闭该文件。
围绕着文件4284/5118，程序抛出了一个MemoryException
但是，任务管理器说内存使用当发生错误时，python.exe * 32是1,854,848K〜= 1.8GB。远远低于我的8 GB限制，或32位程序的4GB限制。
在程序中，我发现内存错误，并报告：
每个列表的长度为4285.
第一个列表包含总共1,928,588,480个float32 〜= 229.9 MB的数据。
第二个列表包含12,342,966,272 uint8的〜= 1,471.3MB的数据。所以，一切似乎都在检查。除了我得到内存错误的部分。
我绝对有更多的内存，它崩溃的文件是〜800KB，所以它没有阅读一个巨大的文件失败。
另外，文件没有损坏。如果我事先没有用完所有的内存，我可以读得很好。为了让事情更加令人困惑，所有这些似乎在我的Linux机器上运行良好（尽管它的内存是16GB而不是Windows机器上的8GB），但是，但是，它似乎并不是导致此问题的机器RAM。
为什么Python会抛出一个内存错误，我希望它能够分配另外2GB的数据？ 我不知道你为什么认为你的程序应该可以访问4GB。根据 Windows版本的内存限制，在MSDN上的64位Windows 7，一个默认的32位进程得到2GB。*哪一个正是它用完的地方。
那么，有没有办法解决这个问题？
那么你可以使用 IMAGE_FILE_LARGE_ADDRESS_AWARE 标志来自定义构建32位Python，并重建 numpy 和所有其他扩展模块。我不能保证所有相关的代码真的可以安全地使用大地址感知标志运行;这是一个很好的机会，但是除非有人已经做了这个测试，否则一个很好的机会是任何人都可能知道的最好的选择。
或者更明显，只需要使用64位的Python。
物理RAM的数量是完全不相关的。你似乎认为你有
8GB内存的8GB限制，但这不是它的工作原理。你的系统把你所有的RAM加上它所需要的交换空间，然后在应用程序之间进行分割。一个应用程序可以获得20GB的虚拟内存，即使在8GB的机器上也不会出现内存错误。与此同时，一个32位应用程序无法访问超过4GB，操作系统将占用一些地址空间（默认情况下，一半在Windows上），所以即使在8GB的机器上也只能获得2GB的地址空间那不是别的（并不是说在一个现代操作系统上可能会不运行任何东西，但你知道我的意思。）
那么，为什么这个工作在你的linux机器上呢？
因为你的linux机器被配置为给予32位进程3.5GB的虚拟地址空间或3.99 GB或者...我不能告诉你确切的数字，但是我见过多年的发行版至少配置了3.25GB。
*另外请注意，您甚至不会真正为您的数据获得完整的2GB;你的程序。大部分的操作系统及其驱动程序可以让你的代码访问到另一半，但是有些位可以放在你的一半，随着你加载的每一个DLL和他们需要的空间以及其他各种东西。它不会加起来太多，但它不是零。
I'm getting a memory issue I can't seem to understand.
I'm on a windows 7 64 bit machine with 8GB of memory and running a 32bit python program.
The programs reads a 5,118 zipped numpy files (npz). Windows reports that the files take up 1.98 GB on disk
Each npz file contains two pieces of data: 'arr_0' is of type np.float32 and 'arr_1' is of type np.uint8
The python script reads each file appends their data into two lists and then closes the file.
Around file 4284/5118 the program throws a MemoryException
However, the task manager says that the memory usage of python.exe *32 when the error occurs is 1,854,848K ~= 1.8GB. Much less than my 8 GB limit, or the supposed 4GB limit of a 32bit program.
In the program I catch the memory error and it reports: Each list has length 4285. The first list contains a total of 1,928,588,480 float32's ~= 229.9 MB of data. The second list contains 12,342,966,272 uint8's ~= 1,471.3MB of data.
So, everything seems to be checking out. Except for the part where I get a memory error. I absolutely have more memory, and the file which it crashes on is ~800KB, so its not failing on reading a huge file.
Also, the file isn't corrupted. I can read it just fine, if I don't use up all that memory beforehand.
To make things more confusing, all of this seems to work fine on my Linux machine (although it does have 16GB of memory as opposed to 8GB on my Windows machine), but still, it doesn't seem to be the machine's RAM that is causing this issue.
Why is Python throwing a memory error, when I expect that it should be able to allocate another 2GB of data?
I don't know why you think your process should be able to access 4GB. According to Memory Limits for Windows Releases at MSDN, on 64-bit Windows 7, a default 32-bit process gets 2GB.* Which is exactly where it's running out.
So, is there a way around this?
Well, you could make a custom build of 32-bit Python that uses the IMAGE_FILE_LARGE_ADDRESS_AWARE flag, and rebuild numpy and all of your other extension modules. I can't promise that all of the relevant code really is safe to run with the large-address-aware flag; there's a good chance it is, but unless someone's already done it and tested it, "a good chance" is the best anyone is likely to know.
Or, more obviously, just use 64-bit Python instead.
The amount of physical RAM is completely irrelevant. You seem to think that you have an "8GB limit" with 8GB of RAM, but that's not how it works. Your system takes all of your RAM plus whatever swap space it needs and divides it up between apps; an app may be able to get 20GB of virtual memory without getting a memory error even on an 8GB machine. And meanwhile, a 32-bit app has no way of accessing more than 4GB, and the OS will use up some of that address space (half of it by default, on Windows), so you can only get 2GB even on an 8GB machine that's not running anything else. (Not that it's possible to ever be "not running anything else" on a modern OS, but you know what I mean.)
So, why does this work on your linux box?
Because your linux box is configured to give 32-bit processes 3.5GB of virtual address space, or 3.99GB, or… Well, I can't tell you the exact number, but every distro I've seen for many years has been configured for at least 3.25GB.
* Also note that you don't even really get that full 2GB for your data; your program. Most of what the OS and its drivers make accessible to your code sits in the other half, but some bits sit in your half, along with every DLL you load and any space they need, and various other things. It doesn't add up to too much, but it's not zero.
这篇关于Python在64位窗口上的32位内存限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
