为什么在64位平台上LuaJIT的内存限制为1-2 GB? [英] Why is LuaJIT's memory limited to 1-2 GB on 64-bit platforms?

问题描述

在64位平台上，LuaJIT仅允许最多1-2GB的数据(不计算使用malloc分配的对象).这种限制从何而来，为什么比32位平台上的限制还要少?

On 64-bit platforms, LuaJIT allows only up to 1-2GB of data (not counting objects allocated with malloc). Where does this limitation come from, and why is this even less than on 32-bit platforms?

推荐答案

LuaJIT设计为使用32位指针.在x64平台上，限制来自使用 mmap 和MAP_32BIT标记.

LuaJIT is designed to use 32-bit pointers. On x64 platforms the limit comes from the use of mmap and the MAP_32BIT flag.

MAP_32BIT(自Linux 2.4.20，2.6起):

MAP_32BIT (since Linux 2.4.20, 2.6):

将映射放入进程地址空间的前2 GB.对于64位程序，此标志仅在x86-64上受支持.它被添加以允许在前2GB内存中的某个位置分配线程堆栈，从而提高某些早期64位处理器上的上下文切换性能.

Put the mapping into the first 2 Gigabytes of the process address space. This flag is supported only on x86-64, for 64-bit programs. It was added to allow thread stacks to be allocated somewhere in the first 2GB of memory, so as to improve context-switch performance on some early 64-bit processors.

基本上使用此标志限制到前31位，而不是顾名思义的前32位.在

Essentially using this flag limits to the first 31-bits, not the first 32-bits as the name suggests. Have a look here for a nice overview of the 1GB limit using MAP_32BIT in the Linux kernel.

即使您可以拥有超过1GB的空间，LuaJIT的作者也解释了为什么这会对性能造成不利影响:

Even if you could have more than 1GB, the LuaJIT author explains why this would be bad for performance:

• 完整的GC比分配本身花费的时间多50％.
• 如果启用了GC，则会使分配时间加倍.
• 要模拟真实的应用程序，在第三轮中将对象之间的链接随机化.这会使GC时间翻倍！

• A full GC takes 50% more time than the allocations themselves.
• If the GC is enabled, it doubles the allocation time.
• To simulate a real application, the links between objects are randomized in the third run. This doubles the GC time!

那只是1GB！现在想象一下使用8GB内存-完整的GC周期将使CPU繁忙长达24秒！ 好的，所以正常模式是使用增量GC.但这只是意味着开销增加了约30％，混合在分配之间，并且每次都将驱逐CPU缓存.基本上，您的应用程序将由GC开销控制，并且您会开始想知道为什么它这么慢....

And that was just for 1GB! Now imagine using 8GB -- a full GC cycle would keep the CPU busy for a whopping 24 seconds! Ok, so the normal mode is to use the incremental GC. But this just means the overhead is ~30% higher, it's mixed in between the allocations and it will evict the CPU cache every time. Basically your application will be dominated by the GC overhead and you'll begin to wonder why it's slow ....

tl; dr版本:请勿在家中尝试. GC需要重写(推迟到LuaJIT 2.1).

tl;dr version: Don't try this at home. And the GC needs a rewrite (postponed to LuaJIT 2.1).

总而言之，1 GB的限制是Linux内核和LuaJIT垃圾收集器的限制.这仅适用于LuaJIT状态内的对象，可以通过使用malloc克服，它将在较低的32位地址空间之外进行分配.另外，可以在32位模式下使用在x64上构建的x86并访问完整的4GB.

To summarize, the 1GB limit is a limitation of the Linux kernel and the LuaJIT garbage collector. This only applies to objects within the LuaJIT state and can be overcome by using malloc, which will allocate outside the lower 32-bit address space. Also, it's possible to use the x86 build on x64 in 32-bit mode and have access the full 4GB.

查看这些链接以获取更多信息:

Check out these links for more information:

• 如何获取Linux上超过64位LuaJIT的1gb内存限制?
• 即使没有MAP_32BIT限制，LuaJIT x64也限制为31位地址空间?
• LuaJIT奇怪的内存限制
• >您从2008年以来从未听说过的最疯狂的错误:Linux线程回归

• How to get past 1gb memory limit of 64 bit LuaJIT on Linux?
• LuaJIT x64 limited to 31 bit address space, even without MAP_32BIT restrictions?
• LuaJIT strange memory limit
• Digging out the craziest bug you never heard about from 2008: a linux threading regression

这篇关于为什么在64位平台上LuaJIT的内存限制为1-2 GB?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！