malloc_trim() 可以从堆中间释放内存吗? [英] Can malloc_trim() release memory from the middle of the heap?

问题描述

我对 glibc 中实现的 malloc_trim 的行为感到困惑.

man malloc_trim[...]malloc_trim - 从堆顶释放空闲内存[...]此函数无法释放位于堆顶部以外的位置的空闲内存.

当我现在查找 malloc_trim() 的源代码(在 malloc/malloc.c 中)时，我看到它调用了 mtrim()，它正在使用 madvise(x, MADV_DONTNEED) 将内存释放回操作系统.

所以我想知道手册页是否错误，或者我是否误解了 malloc/malloc.c 中的源代码.

malloc_trim()可以从堆中间释放内存吗?

解决方案

现在glibc中madvise和MADV_DONTNEED有两种用法:http://code.metager.de/source/搜索?q=MADV_DONTNEED&path=%2Fgnu%2Fglibc%2Fmalloc%2F&project=gnu

 H A D arena.c 643 __madvise ((char *) h + new_size, diff, MADV_DONTNEED);H A D malloc.c 4535 __madvise (paligned_mem, size & ~psm1, MADV_DONTNEED);

有 https://sourceware.org/git/?p=glibc.git;a=commit;f=malloc/malloc.c;h=68631c8eb92ff38d9da1ae34f6aa048539b199cc 由 Ulrich Drepper 于 2007 年 12 月 16 日提交(glibc 2.9 和较新):

<块引用>

• malloc/malloc.c (public_mTRIm):遍历所有 arenas 并调用

mTRIm 适用于所有这些.(mTRIm):另外迭代所有空闲块并使用 madvise为所有包含至少一个块的块释放内存内存页.

mTRIm(现在 mtrim) 实现已更改.块的未使用部分，在页面大小上对齐并且大小超过页面可以标记为MADV_DONTNEED:

/* 查看块是否包含至少一个未使用的页面.*/char *paligned_mem = (char *) (((uintptr_t) p+ sizeof (struct malloc_chunk)+ psm1) &〜psm1)；断言 ((char *) chunk2mem (p) + 4 * SIZE_SZ <= paligned_mem);断言 ((char *) p + 大小 > paligned_mem);/* 这是我们可能释放的大小.*/大小 -= paligned_mem - (char *) p;如果(尺寸 > psm1)madvise (paligned_mem, size & ~psm1, MADV_DONTNEED);

malloc_trim 的手册页在那里:https://github.com/mkerrisk/man-pages/blob/master/man3/malloc_trim.3 并由 kerrisk 在 2012 年提交:https://github.com/mkerrisk/man-pages/commit/a15b0e60b297e29c825b7417582a33e6ca26bf65

我可以 grep glibc 的 git，glibc 中没有手册页，也没有提交到 malloc_trim 手册页来记录这个补丁.glibc malloc 最好也是唯一的文档是它的源代码:https://sourceware.org/git/?p=glibc.git;a=blob;f=malloc/malloc.c

附加功能:malloc_trim(size_t pad);第609章/*610 malloc_trim(size_t 垫)；611612 如果可能，将内存还给系统(通过负sbrk 的 613 个参数)如果在 `high' 端有未使用的内存614 malloc 池.您可以在释放大块615 内存以潜在地降低系统级内存需求616个节目.但是，它不能保证减少内存.在下面617 一些分配模式，一些大的空闲内存块将618 锁定在两个已使用的块之间，因此它们无法归还给619系统.620621 malloc_trim 的 `pad' 参数表示空闲的数量622 尾随空格未修剪.如果这个参数为零，623 只用最小的内存量来维护内部数据将留下 624 个结构(一页或更少).非零参数可以提供 625 以保持足够的尾随空间进行维修626 个未来预期分配，无需重新获取内存627 来自系统.628629 如果 Malloc_trim 实际释放了任何内存，则返回 1，否则返回 0.630 在不支持负 sbrks"的系统上，它总是631 返回 0.632 */第633章634

从块的中间释放没有记录为 malloc/malloc.c 中的文本(并且 malloc_trim 在推荐中的描述没有在 2007 年更新)，也没有记录在手册页项目中.2012 年的手册页可能是该函数的第一个手册页，不是由 glibc 的作者编写的.glibc 的信息页面仅提及 128 KB 的 M_TRIM_THRESHOLD:https://www.gnu.org/software/libc/manual/html_node/Malloc-Tunable-Parameters.html#Malloc-Tunable-Parameters 并且不要列出 malloc_trim 函数 https://www.gnu.org/software/libc/manual/html_node/Summary-of-Malloc.html#Summary-of-Malloc(而且它也不记录 memusage/memusagestat/libmemusage.so).

您可以再次询问 Drepper 和其他 glibc 开发人员，就像您在 https://sourceware.org/ml/libc-help/2015-02/msg00022.html malloc_trim() 行为"，但仍然没有得到他们的回复.(只有其他用户的错误答案，例如 https://sourceware.org/ml/libc-help/2015-05/msg00007.html https://sourceware.org/ml/libc-help/2015-05/msg00008.html)

或者你可以用这个简单的 C 程序 (test_malloc_trim.c) 和 strace/ltracemalloc_trim>:

#include <stdlib.h>#include <stdio.h>#include <unistd.h>#include <malloc.h>主函数(){整数 *m1,*m2,*m3,*m4;printf("%s
","测试开始");m1=(int*)malloc(20000);m2=(int*)malloc(40000);m3=(int*)malloc(80000);m4=(int*)malloc(10000);printf("1:%p 2:%p 3:%p 4:%p
", m1, m2, m3, m4);免费(平方米)；malloc_trim(0);//20000, 2000000睡眠(1)；免费(m1)；免费(立方米)；免费(平方米)；//malloc_stats();malloc_info(0, 标准输出);返回0；}

gcc test_malloc_trim.c -o test_malloc_trim, strace ./test_malloc_trim

write(1, "测试开始
", 13测试开始) = 13brk(0) = 0xcca000brk(0xcef000) = 0xcef000写(1，1:0xcca010 2:0xccee40 3:0xcd8a90"...，441:0xcca010 2:0xccee40 3:0xcd8a90 4:0xcec320) = 44疯狂(0xccf000、36864、MADV_DONTNEED)= 0rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0nanosleep({1, 0}, 0x7ffffafbffff0) = 0brk(0xceb000) = 0xceb000

所以，在 malloc_trim(0) 调用后，有 9 个页面的 madvise 和 MADV_DONTNEED，此时在堆的中间.

I am confused about the behaviour of malloc_trim as implemented in the glibc.

man malloc_trim
[...]
malloc_trim - release free memory from the top of the heap
[...]
This function cannot release free memory located at places other than the top of the heap.

When I now look up the source of malloc_trim() (in malloc/malloc.c) I see that it calls mtrim() which is utilizing madvise(x, MADV_DONTNEED) to release memory back to the operating system.

So I wonder if the man-page is wrong or if I misinterpret the source in malloc/malloc.c.

Can malloc_trim() release memory from the middle of the heap?

解决方案

There are two usages of madvise with MADV_DONTNEED in glibc now: http://code.metager.de/source/search?q=MADV_DONTNEED&path=%2Fgnu%2Fglibc%2Fmalloc%2F&project=gnu

 H A D  arena.c 643 __madvise ((char *) h + new_size, diff, MADV_DONTNEED);
 H A D  malloc.c    4535 __madvise (paligned_mem, size & ~psm1, MADV_DONTNEED);

There was https://sourceware.org/git/?p=glibc.git;a=commit;f=malloc/malloc.c;h=68631c8eb92ff38d9da1ae34f6aa048539b199cc commit by Ulrich Drepper on 16 Dec 2007 (part of glibc 2.9 and newer):

• malloc/malloc.c (public_mTRIm): Iterate over all arenas and call

mTRIm for all of them. (mTRIm): Additionally iterate over all free blocks and use madvise to free memory for all those blocks which contain at least one memory page.

mTRIm (now mtrim) implementation was changed. Unused parts of chunks, aligned on page size and having size more than page may be marked as MADV_DONTNEED:

           /* See whether the chunk contains at least one unused page.  */
           char *paligned_mem = (char *) (((uintptr_t) p
                                           + sizeof (struct malloc_chunk)
                                           + psm1) & ~psm1);

           assert ((char *) chunk2mem (p) + 4 * SIZE_SZ <= paligned_mem);
           assert ((char *) p + size > paligned_mem);

           /* This is the size we could potentially free.  */
           size -= paligned_mem - (char *) p;

           if (size > psm1)
               madvise (paligned_mem, size & ~psm1, MADV_DONTNEED);

Man page of malloc_trim is there: https://github.com/mkerrisk/man-pages/blob/master/man3/malloc_trim.3 and it was committed by kerrisk in 2012: https://github.com/mkerrisk/man-pages/commit/a15b0e60b297e29c825b7417582a33e6ca26bf65

As I can grep the glibc's git, there are no man pages in the glibc, and no commit to malloc_trim manpage to document this patch. The best and the only documentation of glibc malloc is its source code: https://sourceware.org/git/?p=glibc.git;a=blob;f=malloc/malloc.c

Additional functions:
 malloc_trim(size_t pad);
 609 /*
 610   malloc_trim(size_t pad);
 611 
 612   If possible, gives memory back to the system (via negative
 613   arguments to sbrk) if there is unused memory at the `high' end of
 614   the malloc pool. You can call this after freeing large blocks of
 615   memory to potentially reduce the system-level memory requirements
 616   of a program. However, it cannot guarantee to reduce memory. Under
 617   some allocation patterns, some large free blocks of memory will be
 618   locked between two used chunks, so they cannot be given back to
 619   the system.
 620 
 621   The `pad' argument to malloc_trim represents the amount of free
 622   trailing space to leave untrimmed. If this argument is zero,
 623   only the minimum amount of memory to maintain internal data
 624   structures will be left (one page or less). Non-zero arguments
 625   can be supplied to maintain enough trailing space to service
 626   future expected allocations without having to re-obtain memory
 627   from the system.
 628 
 629   Malloc_trim returns 1 if it actually released any memory, else 0.
 630   On systems that do not support "negative sbrks", it will always
 631   return 0.
 632 */
 633 int      __malloc_trim(size_t);
 634 

Freeing from the middle of the chunk is not documented as text in malloc/malloc.c (and malloc_trim description in commend was not updated in 2007) and not documented in man-pages project. Man page from 2012 may be the first man page of the function, written not by authors of glibc. Info page of glibc only mentions M_TRIM_THRESHOLD of 128 KB: https://www.gnu.org/software/libc/manual/html_node/Malloc-Tunable-Parameters.html#Malloc-Tunable-Parameters and don't list malloc_trim function https://www.gnu.org/software/libc/manual/html_node/Summary-of-Malloc.html#Summary-of-Malloc (and it also don't document memusage/memusagestat/libmemusage.so).

You may ask Drepper and other glibc developers again as you already did in https://sourceware.org/ml/libc-help/2015-02/msg00022.html "malloc_trim() behaviour", but there is still no reply from them. (Only wrong answers from other users like https://sourceware.org/ml/libc-help/2015-05/msg00007.html https://sourceware.org/ml/libc-help/2015-05/msg00008.html)

Or you may test the malloc_trim with this simple C program (test_malloc_trim.c) and strace/ltrace:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <malloc.h>

int main()
{
    int *m1,*m2,*m3,*m4;
    printf("%s
","Test started");
    m1=(int*)malloc(20000);
    m2=(int*)malloc(40000);
    m3=(int*)malloc(80000);
    m4=(int*)malloc(10000);
    printf("1:%p 2:%p 3:%p 4:%p
", m1, m2, m3, m4);
    free(m2);
    malloc_trim(0); // 20000, 2000000
    sleep(1);
    free(m1);
    free(m3);
    free(m4);
    // malloc_stats(); malloc_info(0, stdout);
    return 0;
}

gcc test_malloc_trim.c -o test_malloc_trim, strace ./test_malloc_trim

write(1, "Test started
", 13Test started
)          = 13
brk(0)                                  = 0xcca000
brk(0xcef000)                           = 0xcef000
write(1, "1:0xcca010 2:0xccee40 3:0xcd8a90"..., 441:0xcca010 2:0xccee40 3:0xcd8a90 4:0xcec320
) = 44
madvise(0xccf000, 36864, MADV_DONTNEED) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0
rt_sigaction(SIGCHLD, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
nanosleep({1, 0}, 0x7ffffafbfff0)       = 0
brk(0xceb000)                           = 0xceb000

So, there is madvise with MADV_DONTNEED for 9 pages after malloc_trim(0) call, when there was hole of 40008 bytes in the middle of the heap.

这篇关于malloc_trim() 可以从堆中间释放内存吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！