Linux 分配器不释放小块内存 [英] Linux Allocator Does Not Release Small Chunks of Memory

问题描述

Linux glibc 分配器的行为似乎很奇怪.希望有人可以对此有所了解.这是我拥有的源文件:

first.cpp:

#include #include #include <列表>#include <向量>int main() {std::list指针；for(size_t i = 0; i <50000; ++i) {ptrs.push_back(新字符[1024]);}for(size_t i = 0; i <50000; ++i) {删除[] ptrs.back();ptrs.pop_back();}ptrs.clear();睡眠(100)；返回0；}

第二个.cpp:

#include #include #include <列表>int main() {字符** ptr = 新字符*[50000];for(size_t i = 0; i <50000; ++i) {ptrs[i] = 新字符 [1024];}for(size_t i = 0; i <50000; ++i) {删除[] ptrs[i];}删除 [] 指针；睡眠(100)；返回0；}

我都编译:

<前>$ g++ -o first first.cpp$ g++ -o second second.cpp

我先跑，它休眠后，我看常驻内存大小:

当我编译first.cpp，并运行它时，我用ps查看内存:

$ ./first&$ ps 辅助 |首先是grepdavidw 9393 1.3 0.3 64344 53016 pts/4 S 23:37 0:00 ./first$ ./秒&$ ps 辅助 |grep 秒davidw 9404 1.0 0.0 12068 1024 pts/4 S 23:38 0:00 ./秒

注意常驻内存大小.首先，常驻内存大小为 53016k.其次，它是 1024k.首先，出于某种原因，从未将分配释放回内核.

为什么第一个程序不将内存交给内核，而第二个程序却可以?我知道第一个程序使用链表，链表可能在与我们释放的数据相同的页面上分配一些节点.然而，这些节点应该被释放，因为我们正在弹出这些节点，然后清除链表.如果您通过 valgrind 运行这些程序中的任何一个，它就会返回而不会出现内存泄漏.可能发生的事情是内存在 first.cpp 中碎片化，而在 second.cpp 中没有.然而，如果一个页面上的所有内存都被释放了，那该页面如何不被释放回内核呢?将内存交还给内核需要什么?如何修改 first.cpp(继续将 char* 放入列表中)，以便将内存交给内核.

解决方案

这种行为是有意为之，glibc 使用一个可调阈值来决定是否实际将内存返回给系统，或者是否缓存它以供以后重用.在您的第一个程序中，您对每个 push_back 进行了大量小分配，这些小分配不是连续块，并且可能低于阈值，因此不要返回给操作系统.

调用malloc_trim(0) 清除列表后应立即导致 glibc将空闲内存的最顶层区域返回给系统(需要 sbrk 系统调用下次内存需要.)

如果您真的需要覆盖默认行为(除非分析显示它确实有帮助，否则我不建议这样做)，那么您可能应该使用 strace 和/或试验mallinfo看看你的程序实际发生了什么，也许使用 mallopt 到调整内存返还给系统的阈值.

The Linux glibc allocator seems to be behaving weirdly. Hopefully, someone can shed some light on this. Here is the source file that I have:

first.cpp:

#include <unistd.h>
#include <stdlib.h>
#include <list>
#include <vector>

int main() {

  std::list<char*> ptrs;
  for(size_t i = 0; i < 50000; ++i) {
    ptrs.push_back( new char[1024] );
  }
  for(size_t i = 0; i < 50000; ++i) {
    delete[] ptrs.back();
    ptrs.pop_back();
  }

  ptrs.clear();

  sleep(100);

  return 0;
}

second.cpp:

#include <unistd.h>
#include <stdlib.h>
#include <list>

int main() {

  char** ptrs = new char*[50000];
  for(size_t i = 0; i < 50000; ++i) {
    ptrs[i] = new char[1024];
  }
  for(size_t i = 0; i < 50000; ++i) {
    delete[] ptrs[i];
  }
  delete[] ptrs;

  sleep(100);

  return 0;
}

I compile both:

$ g++ -o first first.cpp
$ g++ -o second second.cpp

I run first, and after it's sleeping, I see the resident memory size:

When I compile first.cpp, and run it, I look at memory with ps:

$ ./first&
$ ps aux | grep first
davidw    9393  1.3  0.3  64344 53016 pts/4    S    23:37   0:00 ./first


$ ./second&
$ ps aux | grep second
davidw    9404  1.0  0.0  12068  1024 pts/4    S    23:38   0:00 ./second

Notice the resident memory size. In first, the resident memory size is 53016k. in second, it is 1024k. First never released the allocations back to the kernel for some reason or another.

Why does the first program not relinquish memory to the kernel, but the second program does? I understand that the first program uses a linked list and the linked list probably allocates some nodes on the same page as the data we're freeing. However, those nodes should be freed, as we're popping those nodes off, then clearing the linked list. If you run either of these programs through valgrind, it comes back with no memory leaks. What is probably happening is memory gets fragmented in first.cpp that doesn't in second.cpp. However, if all memory on a page is freed, how does that page not get relinquished back to the kernel? What does it take for memory to get relinquished back to the kernel? How can I modify first.cpp (continuing to put the char*'s in a list) so that the memory is relinquished to the kernel.

解决方案

This behaviour is intentional, there is a tunable threshold that glibc uses to decide whether to actually return memory to the system or whether to cache it for later reuse. In your first program you make lots of small allocations with each push_back and those small allocations are not a contiguous block and are presumably below the threshold, so don't get returned to the OS.

Calling malloc_trim(0) after clearing the list should cause glibc to immediately return the top-most region of free memory to the system (requiring a sbrk system call next time memory is needed.)

If you really need to override the default behaviour (which I wouldn't recommend unless profiling reveals it actually helps) then you should probably use strace and/or experiment with mallinfo to see what's actually happening in your program, and maybe using mallopt to adjust the threshold for returning memory to the system.

这篇关于Linux 分配器不释放小块内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！