内存分配阈值(mmap与malloc) [英] Memory allocation threshold (mmap vs malloc)

问题描述

我想指出我是新来的人，所以我试图尽我所能来理解/解释它.

Id like to point out I'm new to this so I'm trying to understand / explain it best i can.

我基本上是想弄清是否有可能由于我的项目的内存限制而使内存分配保持在阈值以下.

I am basically trying to figure out if its possible to keep memory allocation under a threshold due to memory limitation of my project.

这是当前使用第三方libsodium分配内存的方式:

Here is how memory is allocated currently using third party libsodium:

alloc_region(escrypt_region_t *region, size_t size)
{
    uint8_t *base, *aligned;
#if defined(MAP_ANON) && defined(HAVE_MMAP)
    if ((base = (uint8_t *) mmap(NULL, size, PROT_READ | PROT_WRITE,
#ifdef MAP_NOCORE
                                 MAP_ANON | MAP_PRIVATE | MAP_NOCORE,
#else
                                 MAP_ANON | MAP_PRIVATE,
#endif
                                 -1, 0)) == MAP_FAILED)
        base = NULL; /* LCOV_EXCL_LINE */
    aligned  = base;
#elif defined(HAVE_POSIX_MEMALIGN)
    if ((errno = posix_memalign((void **) &base, 64, size)) != 0) {
        base = NULL;
    }
    aligned = base;
#else
    base = aligned = NULL;
    if (size + 63 < size)
        errno = ENOMEM;
    else if ((base = (uint8_t *) malloc(size + 63)) != NULL) {
        aligned = base + 63;
        aligned -= (uintptr_t) aligned & 63;
    }
#endif
    region->base    = base;
    region->aligned = aligned;
    region->size    = base ? size : 0;

    return aligned;
}

因此，例如，当前当前调用posix_memalign来分配(例如)32mb的内存.32mb超出了我给我的内存上限"(但不会发出内存警告，因为内存容量要大得多，这正是我允许"使用的容量)

So for example, this currently calls posix_memalign to allocate (e.g.) 32mb of memory. 32mb exceeds my 'memory cap' given to me (but does not throw memory warnings as the memory capacity is far greater, its just what I'm 'allowed' to use)

通过一些谷歌搜索，我的印象是我可以使用mmap和虚拟内存.我可以看到上面的函数已经实现了一些mmap，但是从未调用过.

From some googling, I'm under the impression i can either use mmap and virtual memory. I can see that the function above already has some mmap implemented but is never called.

是否可以转换上面的代码，以便我永远不会超过我的 30mb 内存限制?

Is it possible to convert the above code so that i never exceed my 30mb memory limit?

据我了解，如果此分配超出了我的可用内存，它将自动在虚拟内存中分配吗?因此，我可以强迫这种情况发生并假装我的可用空间不足吗?

From my understanding, if this allocation would exceed my free memory, it would automatically allocate in virtual memory? So can i force this to happen and pretend that my free space is lower than available?

感谢您的帮助

更新

/* Allocate memory. */
    B_size = (size_t) 128 * r * p;
    V_size = (size_t) 128 * r * N;
    need   = B_size + V_size;
    if (need < V_size) {
        errno = ENOMEM;
        return -1;
    }
    XY_size = (size_t) 256 * r + 64;
    need += XY_size;
    if (need < XY_size) {
        errno = ENOMEM;
        return -1;
    }
    if (local->size < need) {
        if (free_region(local)) {
            return -1;
        }
        if (!alloc_region(local, need)) {
            return -1;
        }
    }
    B  = (uint8_t *) local->aligned;
    V  = (uint32_t *) ((uint8_t *) B + B_size);
    XY = (uint32_t *) ((uint8_t *) V + V_size);

推荐答案

我基本上是想弄清是否有可能由于我的项目的内存限制而使内存分配保持在阈值以下.

I am basically trying to figure out if its possible to keep memory allocation under a threshold due to memory limitation of my project.

在Linux或POSIX系统上，您可以考虑使用 setrlimit(2)与 RLIMIT_AS :

On Linux or POSIX systems, you might consider using setrlimit(2) with RLIMIT_AS:

         This is the maximum size of the process's virtual memory
          (address space) in bytes.  This limit affects calls to brk(2),
          mmap(2), and mremap(2), which fail with the error ENOMEM upon
          exceeding this limit.  

超出此限制， mmap 将失败，因此例如对

Above this limit, the mmap would fail, and so would fail for instance the call to malloc(3) that triggered that particular use of mmap.

我对我可以使用mmap印象很深刻

I'm under the impression i can either use mmap

请注意 malloc(3)将调用 mmap(2)(有时是sbrk(2)...) 从内核中检索(虚拟)内存，从而增加您的虚拟地址空间.但是， malloc 通常更喜欢重用以前的 free -d内存(如果可用).而且 free 通常不会调用 munmap(2)释放内存块，但更喜欢保留它以供将来 malloc -s使用.实际上，大多数C标准库都在小"和大"分配之间进行分隔(实际上，一千兆字节的 malloc 将使用 mmap ，而相应的 free 将立即 mmap ).

Notice that malloc(3) will call mmap(2) (or sometimes sbrk(2)...) to retrieve (virtual) memory from the kernel, thus growing your virtual address space. However, malloc often prefer to reused previously free-d memory (when available). And free usually won't call munmap(2) to release memory chunks but prefer to keep it for future malloc-s. Actually most C standard libraries segregate between "small" and "large" allocation (in practice a malloc for a gigabyte will use mmap, and the corresponding free would mmap immediately).

另请参见 mallopt(3)和 madvise(2).如果您需要将某些页面(由 mmap 获取)锁定到物理 RAM中，请考虑

See also mallopt(3) and madvise(2). In case you need to lock some pages (obtained by mmap) into physical RAM, consider mlock(2).

还要查看 this 答案(说明特定进程使用的 RAM 概念并不那么容易).

Look also into this answer (explaining that the notion of RAM used by a particular process is not that easy).

使用与 malloc 相关的错误(包括内存泄漏) valgrind .

For malloc related bugs (including memory leaks) use valgrind.

这篇关于内存分配阈值(mmap与malloc)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！