在64位系统上以低地址分配内存的最可靠/可移植的方法是什么? [英] What is the most reliable / portable way to allocate memory at low addresses on 64-bit systems?

问题描述

我需要分配大的内存块(供我的自定义分配器使用)，该内存块位于虚拟地址空间的前32GB之内.

I need to allocate large blocks of memory (to be used by my custom allocator) that fall within the first 32GB of virtual address space.

我想像一下，如果我需要1MB的块，我可以从低地址开始使用mmap和MAP_FIXED_NOREPLACE(或VirtualAlloc)以低至1MB的增量进行迭代，直到调用成功.从上一个成功的块继续进行下一个.

I imagine that if I needed, say, 1MB blocks, I could iterate using mmap and MAP_FIXED_NOREPLACE (or VirtualAlloc) from low addresses onwards in increments of, say, 1MB, until the call succeeds. Continue from the last successful block for the next one.

这听起来很笨拙，但至少在抵御OS地址空间布局更改和ASLR算法更改方面会有些健壮.根据我对当前操作系统布局的了解，以这种方式在前32GB中应该有足够的可用内存，但是也许我遗漏了一些东西?

This sounds clumsy, but at least it will be somewhat robust against OS address space layout changes, and ASLR algorithm changes. From my understanding of current OS layouts, there should be plenty of memory available in the first 32GB this way, but maybe I am missing something?

Windows，Linux，OS X，iOS或Android中是否有任何东西可以击败该计划?有更好的方法吗?

Is there anything in Windows, Linux, OS X, iOS or Android that would defeat this scheme? Is there a better way?

仅在您想知道的情况下，这是针对针对一种编程语言的VM的实现，其中在64位系统上将所有指针拟合为32位值可能会带来巨大的内存使用优势，甚至可以提高速度.由于所有对象都至少对齐了8个字节，因此可以移出低3位，从而将指针范围从4GB扩展到32GB.

Just in case you're wondering, this is for the implementation of a VM for a programming language where fitting all pointers in a 32-bit value on a 64-bit system could give huge memory usage advantages and even speed gains. Since all objects are at least 8-byte aligned, the lower 3 bits can be shifted out, expanding the pointer range from 4GB to 32GB.

推荐答案

为了限制Windows中分配的内存范围，我们可以使用

for restrict allocated memory range in windows we can use NtAllocateVirtualMemory function - this api is avaible for use in both user and kernel mode. in user mode it exported by ntdll.dll (use ntdll.lib or ntdllp.lib from wdk). in this api exist parameter - ZeroBits - The number of high-order address bits that must be zero in the base address of the section view. but in msdn link next words about ZeroBits is incorrect. correct is:

零位

提供在以下位置必须为零的高阶地址位的数量 剖视图的基地址.此参数的值必须 小于或等于零位的最大数量，并且仅 在内存管理确定在何处分配视图时使用 (即BaseAddress为null时).

Supplies the number of high order address bits that must be zero in the base address of the section view. The value of this argument must be less than or equal to the maximum number of zero bits and is only used when memory management determines where to allocate the view (i.e. when BaseAddress is null).

如果 ZeroBits 为零，则不应用零位约束.

If ZeroBits is zero, then no zero bit constraints are applied.

如果 ZeroBits 大于0且小于32，则为 位31开头的零位的数目.还需要位63:32 为零.这保留了与32位系统的兼容性. 如果 ZeroBits 大于32，则将其视为掩码，然后算出前导零的数量 在面具里.然后，它成为零位参数.

If ZeroBits is greater than 0 and less than 32, then it is the number of leading zero bits from bit 31. Bits 63:32 are also required to be zero. This retains compatibility with 32-bit systems. If ZeroBits is greater than 32, then it is considered as a mask and then number of leading zero are counted out in the mask. This then becomes the zero bits argument.

因此，实际上我们可以使用 ZeroBits 作为掩码-这是最省电的方式.但可以用作31位的零位计数(在这种情况下，63-32位将始终等于0).因为分配粒度(当前为64kb-0x10000)-低16位始终为0.因此，在位数模式下 ZeroBits 的有效值-从1到15(= 31-16).为了更好地了解此参数的工作原理，请查找示例代码.为了获得更好的演示效果，我将使用

so really we can use ZeroBits as mask - this is most power usage. but can be use and as zero bit count from 31 bit (in this case 63-32 bits will be always equal to 0). because allocation granularity (currently 64kb - 0x10000) - low 16 bits is always 0. so valid value for ZeroBits in bits-number mode - from 1 to 15 (=31-16). for better understand how this parameter work - look for example code. for better demo effect i will be use

MEM_TOP_DOWN

应在最高虚拟地址处创建指定的区域 可能基于 ZeroBits .

The specified region should be created at the highest virtual address possible based on ZeroBits.

PVOID BaseAddress;
ULONG_PTR ZeroBits;
SIZE_T RegionSize = 1;
NTSTATUS status;

for (ZeroBits = 0xFFFFFFFFFFFFFFFF;;)
{
    if (0 <= (status = NtAllocateVirtualMemory(NtCurrentProcess(), &(BaseAddress = 0), 
        ZeroBits, &RegionSize, MEM_RESERVE|MEM_TOP_DOWN, PAGE_NOACCESS)))
    {
        DbgPrint("%p:%p\n", ZeroBits, BaseAddress);
        NtFreeVirtualMemory(NtCurrentProcess(), &BaseAddress, &RegionSize, MEM_RELEASE);

        ZeroBits >>= 1;
    }
    else
    {
        DbgPrint("%x\n", status);
        break;
    }
}

for(ZeroBits = 0;;) 
{
    if (0 <= (status = NtAllocateVirtualMemory(NtCurrentProcess(), &(BaseAddress = 0), 
        ZeroBits, &RegionSize, MEM_RESERVE|MEM_TOP_DOWN, PAGE_NOACCESS)))
    {
        DbgPrint("%x:%p\n", ZeroBits++, BaseAddress);
        NtFreeVirtualMemory(NtCurrentProcess(), &BaseAddress, &RegionSize, MEM_RELEASE);
    }
    else
    {
        DbgPrint("%x\n", status);
        break;
    }
}

并输出:

FFFFFFFFFFFFFFFF:00007FF735B40000
7FFFFFFFFFFFFFFF:00007FF735B40000
3FFFFFFFFFFFFFFF:00007FF735B40000
1FFFFFFFFFFFFFFF:00007FF735B40000
0FFFFFFFFFFFFFFF:00007FF735B40000
07FFFFFFFFFFFFFF:00007FF735B40000
03FFFFFFFFFFFFFF:00007FF735B40000
01FFFFFFFFFFFFFF:00007FF735B40000
00FFFFFFFFFFFFFF:00007FF735B40000
007FFFFFFFFFFFFF:00007FF735B40000
003FFFFFFFFFFFFF:00007FF735B40000
001FFFFFFFFFFFFF:00007FF735B40000
000FFFFFFFFFFFFF:00007FF735B40000
0007FFFFFFFFFFFF:00007FF735B40000
0003FFFFFFFFFFFF:00007FF735B40000
0001FFFFFFFFFFFF:00007FF735B40000
0000FFFFFFFFFFFF:00007FF735B40000
00007FFFFFFFFFFF:00007FF735B40000
00003FFFFFFFFFFF:00003FFFFFFF0000
00001FFFFFFFFFFF:00001FFFFFFF0000
00000FFFFFFFFFFF:00000FFFFFFF0000
000007FFFFFFFFFF:000007FFFFFF0000
000003FFFFFFFFFF:000003FFFFFF0000
000001FFFFFFFFFF:000001FFFFFF0000
000000FFFFFFFFFF:000000FFFFFF0000
0000007FFFFFFFFF:0000007FFFFF0000
0000003FFFFFFFFF:0000003FFFFF0000
0000001FFFFFFFFF:0000001FFFFF0000
0000000FFFFFFFFF:0000000FFFFF0000
00000007FFFFFFFF:00000007FFFF0000
00000003FFFFFFFF:00000003FFFF0000
00000001FFFFFFFF:00000001FFFF0000
00000000FFFFFFFF:00000000FFFF0000
000000007FFFFFFF:000000007FFF0000
000000003FFFFFFF:000000003FFF0000
000000001FFFFFFF:000000001FFF0000
000000000FFFFFFF:000000000FFF0000
0000000007FFFFFF:0000000007FF0000
0000000003FFFFFF:0000000003FF0000
0000000001FFFFFF:0000000001FF0000
0000000000FFFFFF:0000000000FF0000
00000000007FFFFF:00000000007F0000
00000000003FFFFF:00000000003F0000
00000000001FFFFF:00000000001F0000
00000000000FFFFF:00000000000F0000
000000000007FFFF:0000000000070000
000000000003FFFF:0000000000030000
000000000001FFFF:0000000000010000
c0000017
0:00007FF735B40000
1:000000007FFF0000
2:000000003FFF0000
3:000000001FFF0000
4:000000000FFF0000
5:0000000007FF0000
6:0000000003FF0000
7:0000000001FF0000
8:0000000000FF0000
9:00000000007F0000
a:00000000003F0000
b:00000000001F0000
c:00000000000F0000
d:0000000000070000
e:0000000000030000
f:0000000000010000
c0000017

因此，如果我们说想将内存分配限制为32Gb(0x800000000)-我们可以使用ZeroBits = 0x800000000 - 1:

so if we say want restrict memory allocation to 32Gb(0x800000000) - we can use ZeroBits = 0x800000000 - 1:

NtAllocateVirtualMemory(NtCurrentProcess(), &(BaseAddress = 0), 
            0x800000000 - 1, &RegionSize, MEM_RESERVE|MEM_TOP_DOWN, PAGE_NOACCESS)

结果内存将在范围[0, 7FFFFFFFF]中分配(实际上是[0, 7FFFF0000]，因为地址的分配粒度低16位始终为0)

as result memory will be allocated in range [0, 7FFFFFFFF] (actually [0, 7FFFF0000] because allocation granularity low 16 bits of address always is 0)

您可以通过

than you can create heap by RtlCreateHeap on allocated region range and allocate memory from this heap (note - this is also user mode api - use ntdll[p].lib for linker input)

PVOID BaseAddress = 0;
SIZE_T RegionSize = 0x10000000;// reserve 256Mb
if (0 <= NtAllocateVirtualMemory(NtCurrentProcess(), &BaseAddress, 
    0x800000000 - 1, &RegionSize, MEM_RESERVE, PAGE_READWRITE))
{
    if (PVOID hHeap = RtlCreateHeap(0, BaseAddress, RegionSize, 0, 0, 0))
    {
        HeapAlloc(hHeap, 0, <somesize>);
        RtlDestroyHeap(hHeap);
    }

    VirtualFree(BaseAddress, 0, MEM_RELEASE);
}

这篇关于在64位系统上以低地址分配内存的最可靠/可移植的方法是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！